Lightweight Processes with Camunda and Zeebe
With the transition from Camunda 7 to 8, the process engine transforms from a library that can be integrated into applications to a comprehensive platform with many components. While exploring the recommended setup, we encounter Helm charts and deployments with eight or more components, some of which are only available for enterprise customers in production. This article demonstrates how lightweight process applications can be created using the open-source core component Zeebe.
Table of Contents
A New Architecture
It's rare for a new major version of software to bring about such a significant paradigm shift: while the Camunda Platform in version 7 was fully integrable into Java applications, Camunda 8 is now based on Zeebe, an independent component with which our code communicates via gRPC. Instead of representing the state of business processes in a database, the operations of the processes are stored in an event log. The goal of the Camunda developers was to create a process engine that is infinitely scalable horizontally. As a result, Camunda 8 can process a significantly higher throughput than Camunda 7, which could only handle as many requests as its central, relational database.
The Complexity of Camunda Platform 8
A frequently discussed point about the new platform is that the complexity of an average Camunda installation is higher than it was for the old engine. A productive deployment for Camunda 7 included the engine itself, a relational database, optionally a task list for processing user tasks, and a cockpit for inspecting running processes. Except for the database, we could bundle all these components in a single Java application, which we could deploy using familiar methods.
Since the "new" Zeebe engine does not store the state of process instances in a database but is only responsible for orchestrating the associated events and delegating them to job workers who perform the actual processing with code, there is no API to query the state of the process instances. It is up to us to reconstruct the state from the event logs and make it queryable. Tools for this are the so-called exporters, plugins in the Zeebe engine, which transmit the events to Kafka, Redis, Elasticsearch, or other storage technologies where they can be aggregated. This approach is used by Camunda Platform 8 itself: The data used by the task list and the "Cockpit" successor "Operate" are stored in an Elasticsearch repository, which is part of the standard deployment. These components all require maintenance and care, not to mention the consumption of resources.
Another downside: The graphical interfaces "Tasklist" and "Operate" are no longer under an open-source license and may not be used in production without a license (and thus for free) (although we generally would not recommend operating business-critical open-source components in production without a maintenance contract).
A Process Engine for Large Corporations?
The changes made with the Camunda 7 Community Edition have caused uncertainty among its users. Not everyone has technical requirements for a process engine that justify managing so many components and maintaining a Kubernetes cluster. The alternative use of the in-house SaaS product, Camunda Cloud, might make sense for smaller companies, but here the costs scale with the executed process instances, so a use case should be carefully calculated. The break-even point, at which self-hosting becomes worthwhile, is likely only surpassed with a considerable number of process instances per hour.
But what about smaller projects that could benefit from process orchestration and the use of BPMN, but would no longer be profitable with the use of the SaaS offering or the operation of more complex infrastructure? In the following example, we will explore this question by building and operating a lightweight process.
Example: A Reminder Bot for Social Networks
A "Reminder-Bot" is a great example of a lightweight process. It involves an account that is mentioned in a message with the addition "in x days," and after this period, responds with a post as a reminder. As a BPMN process, this could look something like this:
From the process model, the desired process flow can be transparently read: We wait for a user to request a reminder and then start the process. First, we try to interpret the message. Can we understand a time period from the text, or not? If this is not the case, we respond with a message to the original post indicating that we could not understand any instructions. However, if we were able to understand, we briefly confirm the set reminder and then wait for the calculated date.
Now, one of two scenarios can occur: Either the desired date arrives and we write the reminder as requested. Or we are instructed to "cancel" the reminder. In this case, we also confirm this and end the process in a different target status.
Operation of the Process Engine
To illustrate the parameters and required file paths or volumes, we have derived a Docker-Compose file from the Camunda documentation, which contains nothing but Zeebe:
services:
zeebe: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
image: camunda/zeebe:8.4.1
container_name: zeebe
ports:
- "26500:26500"
- "9600:9600"
environment:
- ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
- ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
- "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
restart: always
healthcheck:
test: [ "CMD-SHELL", "timeout 10s bash -c ':> /dev/tcp/127.0.0.1/9600' || exit 1" ]
interval: 30s
timeout: 5s
retries: 5
start_period: 30s
volumes:
- ./zeebe-data:/usr/local/zeebe/dataFor the container, two ports are mapped: 26500 for gRPC communication and 9600 for the startup and ready check from outside. A volume maps the data from /usr/local/zeebe/data to the host folder zeebe-data, where the broker's event log is stored. From the memory configuration -Xmx512m, which defines a maximum heap of 512MB, we can see that the engine does not have high memory requirements. However, for use cases with higher volume, this might not be sufficient. The broker is started with a docker-compose up command.
Project Setup with Spring Boot
We use Spring Boot for our server-side Java application, which interacts with the Zeebe broker. As a Maven dependency, we add the Camunda Spring Boot Starter, which configures the job workers for us and controls their lifecycle:
<dependency>
<groupId>io.camunda.spring</groupId>
<artifactId>spring-boot-starter-camunda</artifactId>
<version>8.4.0</version>
</dependency>In the application.properties file, we define the URL of our Zeebe broker. Since we are in a development environment, we need to disable SSL with the plaintext property - in a production environment, we would recommend setting up a connection over TLS:
zeebe.client.broker.gateway-address=localhost:26500
zeebe.client.security.plaintext=trueWith the help of this Spring Boot integration, we can also define which process instances should be deployed from the classpath when our application starts. This is done using the @Deployment annotation, which we can add to any configuration classes.
@Configuration
@Deployment(resources = "reminder-process.bpmn")
public class ZeebeConfig {
}In our example, we specify the filename of our BPMN definition directly. However, it's also possible to use wildcards to include, for example, all BPMN files or certain files with a specific naming scheme.
Starting the Reminder Process
We want to start a new process instance as soon as our account is mentioned in a posting. To interact with the engine, we need an instance of ZeebeClient, which we can receive through dependency injection thanks to the Spring Boot integration. In the example, we define the Java record ProcessInput, which contains the required input variables for starting a process instance.
@Autowired
private final ZeebeClient zeebeClient;
record ProcessInput(String content,
String statusId,
String visibility,
String account) {};
public Long startReminderProcess(Status status) {
var processInput = new ProcessInput(status.content(),
status.id(),
status.visibility(),
status.account().acct());
var result = zeebeClient
.newCreateInstanceCommand()
.bpmnProcessId("Process_Reminder")
.latestVersion()
.variables(processInput)
.send()
.join();
return result.getProcessInstanceKey();
}Guided by a multi-stage builder, we now create a process instance, which is derived from the process definition with the ID Process_Reminder. We want to start this in the latest deployed version and pass the mapped input variables. The call to send() sends the command asynchronously to the Zeebe Engine via gRPC; with join(), we wait for the successful execution to use the ID of the newly created process instance as the return value of the method.
Executing Program Logic with Job Workers
For our business process to be more than just a visual diagram and actually orchestrate our application, we need to define a type at service tasks. This type is used by so-called job workers to identify the type of task to be processed.
In the following example of a Java job worker, we use the @JobWorker annotation from the previously mentioned Spring Boot Starter to establish the connection to the service task. In the background, our application polls the engine via gRPC to determine if there are any jobs to be processed.
public record ParseRequestResponse(boolean dateUnderstood,
ZonedDateTime reminderDate) {};
@JobWorker(type = "parseDate")
public ParseRequestResponse parseRequest(
@Variable String content) {
var matcher = PATTERN.matcher(content);
if(matcher.find()) {
var result = matcher.group(1);
return new ParseRequestResponse(true,
ZonedDateTime.now()
.plusDays(Long.parseLong(result)));
} else {
return new ParseRequestResponse(false, null);
}
}When this is the case, our method is called. The parameter String content listed in the signature is derived from the process variables and passed to the method. We attempt to extract a mention of days using regular expressions from the content and add these days to the current date to determine the date for the reminder. The response to the process engine is returned as the method's return value. In our case, we have defined a local Java record that contains two variables: a boolean value that represents whether the request was understood at all, and a ZonedDateTime, which holds the calculated date of the reminder. The Camunda Spring integration maps each field of the response object to a process variable after the method call, mapping the field names to variable names. Thus, in this example, we introduce two new process variables: dateUnderstood and reminderDate.
The principle is very similar when directly interacting with posts on the social network. Here too, we register a job worker with the engine, which listens to the assigned task. When this task is reached by the engine within a process instance, the method annotated with @JobWorker is called, and the variables are mapped into the method's parameters.
public record SuccessOutput(String reminderStatusId) {};
@JobWorker(type = "confirmCancellation")
public SuccessOutput confirmCancellation(
@Variable String statusId,
@Variable String visibility,
@Variable String account) {
var message = MESSAGE_TEMPLATE.formatted(account,
"Na gut, der Timer ist abgebrochen");
var result = mastodonService.replyToStatus(statusId,
message,
visibility);
return new SuccessOutput(result.id());
}This worker receives the information on which status to reply to via statusId, the visibility in which the status was written via visibility, and to which account to reply with account. The status ID is needed to attach the post to the correct predecessor, the account to mention the user in it, and the visibility of the original post to reply in the same visibility - after all, we don't want to respond publicly visible to a private mention. With this information, we assemble the message from the (not listed) message template and send it to a server of the decentralized social network.
The same applies to the job workers for the other tasks, which are not described in detail here, but are listed for the sake of completeness.
@JobWorker(type = "confirmReminder")
public SuccessOutput confirmReminder(
@Variable String statusId,
@Variable ZonedDateTime reminderDate,
@Variable String visibility,
@Variable String account) {
var message = MESSAGE_TEMPLATE.formatted(account,
"Hey, alles klar! Ich erinnere dich am %s".formatted(reminderDate));
var result = mastodonService.
replyToStatus(statusId, message, visibility);
return new SuccessOutput(result.id());
}
@JobWorker(type = "respondDateNotUnderstood")
public void replyNotUnderstood(
@Variable String statusId,
@Variable String visibility,
@Variable String account) {
var message = MESSAGE_TEMPLATE.formatted(account,
"Hi, das habe ich leider nicht verstanden. Schreibe mir 'in x Tagen' oder einfach 'x Tage'");
mastodonService.replyToStatus(statusId, message, visibility);
}
@JobWorker(type = "remindUser")
public void remind(@Variable String statusId,
@Variable String visibility,
@Variable String account) {
var message = MESSAGE_TEMPLATE.formatted(account, "Hi, hier ist deine Erinnerung!");
mastodonService.replyToStatus(statusId, message, visibility);
}Limitations
From the example, we can see that smaller applications can be implemented with the "new" Camunda engine without operating a Kubernetes cluster for the infrastructure. Besides our program logic in a Spring Boot application, we initially only need the Zeebe broker, which could also serve several applications simultaneously. However, an important element of process engines is missing in our example: Insight into the running process engine. How can we see which process instances are in what state, or view and resolve incidents? With our setup: Not at all. As mentioned at the beginning, it is not possible to query current states from the Zeebe broker via gRPC; we can only react to events from the engine. In the full Camunda 8 platform, these events are transmitted to an Elasticsearch instance using the Elasticsearch exporter, which manages a queryable state. The Tasklist and the Operate application (formerly: Cockpit), which are based on this state, can only be used productively with a paid enterprise license. From within the community, the Simple Zeebe Monitor was developed as an open solution, providing basic functions for insight and interaction with process instances. The state of the process instances is stored in Kafka, Hazelcast, or Redis for this purpose and must also be transmitted with an appropriate Zeebe exporter. So, we again need additional infrastructure for this.
Conclusion
Even for smaller software projects, the new process engine behind Camunda 8, Zeebe, can be of interest. We certainly do not have to manage the complexity of the full Camunda 8 deployment. Nor is it necessary to use the Camunda Cloud or acquire a Camunda 8 Enterprise license to use Zeebe. If we are willing to maintain some additional infrastructure, we can even enable monitoring and administration of process instances with the Simple Zeebe Monitor. While this may be a viable path for small projects, hobby or open-source applications, it is less likely to be suitable for the migration of Camunda 7 applications that are in productive use in companies under the Community Edition or use user tasks.
About the author
