Messaging and Message Brokers
In standard web-based frameworks, communication works via HTTP requests. The client (e.g. the frontend of a website) makes a GET or POST to server (backend) and receives a response. This standard framework, however, exposes your backend to a lot of security issues and it takes a lot of effort making it secure.
DASF uses a slightly different approach. Here, client and backend cannot communicate with each other directly. Instead, we use a so-called message broker. The client stub (see above) sends the request to a so-called message broker which forwards the request to the server stub.
Then the server processes the request and sends back a response to the client, again via the message broker.
It is somehow comparable to a chat system. Assume Alice wants to chat with Bob using their mobile phones. The mobile phones do not know each other but instead they both register at a server (the message broker). When Alice wants to send a message to Bob, she actually sends it to the server and the server forwards it to Bobs mobile phone.
The advantage of such a framework is that neither the client nor the server need to be accessible from the web. This is especially important if you want to use protected computing resources such as internal servers or HPC clusters. The only thing you have to do is to protect and secure the message broker, this already takes most of the burden to secure your backend module.
Message brokers distinguish between consumers and producers. As the names suggest, producers produce a message and consumers consume the message. When a client stub makes a request, the client is the producer, and the server is the consumer.
And for the response, the client is the consumer and the server is the producer.
To connect producer and consumer, we use so-called topics. You can think of a topic to be the address of the consumer at the message broker. A client that wants to send a request to a backend module, needs to know the topic that the backend module is listening to (i.e. that the backend module is consuming). Likewise, the backend module needs to know where it should send the response to (i.e. it needs to know the topic that the client is consuming).
Within the DASF Framework, you specify the topic when you start listening for
incoming messages. This is specified with the
python ExampleMessageConsumer.py -t my-personal-topic listen
or you specify it in the call of the
if __name__ == "__main__":
The response topic, i.e. the topic that the client is consuming, is specified
within the properties of the request (see
From this, the backend module automatically knows where to send the response
Choosing the message broker
We aim to be agnostic to what message broker is used. Currently, however, we
only tested our workflows with the Apache Pulsar and our Django-based message
broker. An important point is that, by design, our connection to the message
broker does not use provided client libraries, such as the
etc.. Instead, we use the websocket protocol to connect to the message broker.
This is necessary to guarantee flexibility of the framework. So the choice of
your message broker depends on the application that you want to build.