Messaging and Message Brokers

Messaging

In standard web-based frameworks, communication works via HTTP requests. The client (e.g. the frontend of a website) makes a GET or POST to server (backend) and receives a response. This standard framework, however, exposes your backend to a lot of security issues and it takes a lot of effort making it secure.

DASF uses a slightly different approach. Here, client and backend cannot communicate with each other directly. Instead, we use a so-called message broker. The client stub (see above) sends the request to a so-called message broker which forwards the request to the server stub.

Request to the backend module

Request to the backend module

Then the server processes the request and sends back a response to the client, again via the message broker.

Response from the backend module

Response from the backend module

It is somehow comparable to a chat system. Assume Alice wants to chat with Bob using their mobile phones. The mobile phones do not know each other but instead they both register at a server (the message broker). When Alice wants to send a message to Bob, she actually sends it to the server and the server forwards it to Bobs mobile phone.

The advantage of such a framework is that neither the client nor the server need to be accessible from the web. This is especially important if you want to use protected computing resources such as internal servers or HPC clusters. The only thing you have to do is to protect and secure the message broker, this already takes most of the burden to secure your backend module.

Message Brokers

Concept

Message brokers distinguish between consumers and producers. As the names suggest, producers produce a message and consumers consume the message. When a client stub makes a request, the client is the producer, and the server is the consumer.

Request to the backend module

Request to the backend module. The client produces a message, the server consumes.

And for the response, the client is the consumer and the server is the producer.

Response from the backend module

Response from the backend module. The server produces a message, the client consumes.

To connect producer and consumer, we use so-called topics. You can think of a topic to be the address of the consumer at the message broker. A client that wants to send a request to a backend module, needs to know the topic that the backend module is listening to (i.e. that the backend module is consuming). Likewise, the backend module needs to know where it should send the response to (i.e. it needs to know the topic that the client is consuming).

Within the DASF Framework, you specify the topic when you start listening for incoming messages. This is specified with the -t option

python ExampleMessageConsumer.py -t my-personal-topic listen

or you specify it in the call of the main() function,

if __name__ == "__main__":
    main(messaging_config=dict(topic="my-personal-topic"))

The response topic, i.e. the topic that the client is consuming, is specified within the properties of the request (see demessaging.messaging.constants.PropertyKeys). From this, the backend module automatically knows where to send the response to.

Choosing the message broker

There are a variety of message brokers around, such as Apache Pulsar or MQTT. And we are working on a Django-based message broker in the DASF context.

We aim to be agnostic to what message broker is used. Currently, however, we only tested our workflows with the Apache Pulsar and our Django-based message broker. An important point is that, by design, our connection to the message broker does not use provided client libraries, such as the pulsar-client, etc.. Instead, we use the websocket protocol to connect to the message broker. This is necessary to guarantee flexibility of the framework. So the choice of your message broker depends on the application that you want to build.