{"id":107682,"date":"2019-11-19T08:00:53","date_gmt":"2019-11-19T08:00:53","guid":{"rendered":"https:\/\/developers.redhat.com\/blog\/?p=648217"},"modified":"2019-11-19T08:00:53","modified_gmt":"2019-11-19T08:00:53","slug":"decoupling-microservices-with-apache-camel-and-debezium","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2019\/11\/19\/decoupling-microservices-with-apache-camel-and-debezium\/","title":{"rendered":"Decoupling microservices with Apache Camel and Debezium"},"content":{"rendered":"

The rise of microservices<\/a>-oriented architecture brought us new development paradigms and mantras about independent development and decoupling. In such a scenario, we have to deal with a situation where we aim for independence, but we still need to react to state changes in different enterprise domains.<\/p>\n

I’ll use a simple and typical example in order to show what we’re talking about. Imagine the development of two independent microservices: Order<\/code> and User<\/code>. We designed them to expose a REST interface and to each use a separate database, as shown in Figure 1:<\/p>\n

\"Diagram<\/p>\n

Figure 1: Order and User microservices.<\/p>\n<\/div>\n

<\/span><\/p>\n

We must notify the User<\/code> domain about any change happening in the Order<\/code> domain. To do this in the example, we need to update the order_list<\/code>. For this reason, we’ve modeled the User REST service with\u00a0addOrder<\/code> and\u00a0deleteOrder<\/code> operations.<\/p>\n

Solution 1: Queue decoupling<\/h2>\n

The first solution to consider is adding a queue between the services. Order<\/code> will publish events that\u00a0User<\/code> will eventually process, as shown in Figure 2:<\/p>\n

\"Diagram<\/p>\n

Figure 2: Decoupling with a queue.<\/p>\n<\/div>\n

This is a fair design. However, if you don’t use the right middleware you will mix a lot of infrastructure code<\/em> into your domain logic. <\/em>Now that you have queues, you must develop producer<\/em> and consumer logic<\/em>. You also have to take care of transactions. <\/em>The problem is to make sure that every event ends up correctly in both the Order<\/code> database and in the queue.<\/p>\n

Solution 2: Change data capture decoupling<\/h2>\n

Let me introduce an alternative solution that handles all of that work without your touching any line of your microservices code. I’ll use Debezium<\/a> and Apache Camel<\/a>\u00a0to capture data changes on Order<\/code> and trigger certain actions on User<\/code>. Debezium is a log-based data change capture middleware. Camel is an integration framework that simplifies the integration between a source (Order<\/code>) and a destination (User<\/code>), as shown in Figure 3:<\/p>\n

\"Diagram<\/p>\n

Figure 3: Decoupling with Debezium and Camel.<\/p>\n<\/div>\n

Debezium is in charge of capturing any data change happening in the Order<\/code> domain and publishing it to a topic<\/em>. Then a Camel consumer can pick that event and make a REST call to the User<\/code> API to perform the necessary action expected by its domain (in our simple case, update the list).<\/p>\n

Decoupling with Debezium and Camel<\/h2>\n

I’ve prepared a simple demo with all of the components we need to run the example above. You can find this demo\u00a0in this GitHub repo<\/a>. The only part we need to develop is represented by the following source code:<\/p>\n

public class MyRouteBuilder extends RouteBuilder { public void configure() { from(\"debezium:mysql?name=my-sql-connector\" + \"&databaseServerId=1\" + \"&databaseHostName=localhost\" + \"&databaseUser=debezium\" + \"&databasePassword=dbz\" + \"&databaseServerName=my-app-connector\" + \"&databaseHistoryFileName=\/tmp\/dbhistory.dat\" + \"&databaseWhitelist=debezium\" + \"&tableWhitelist=debezium._order\" + \"&offsetStorageFileName=\/tmp\/offset.dat\") .choice() .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo(\"c\")) .process(new AfterStructToOrderTranslator()) .to(\"rest-swagger:http:\/\/localhost:8082\/v2\/api-docs#addOrderUsingPOST\") .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo(\"d\")) .process(new BeforeStructToOrderTranslator()) .to(\"rest-swagger:http:\/\/localhost:8082\/v2\/api-docs#deleteOrderUsingDELETE\") .log(\"Response : ${body}\"); } } <\/pre>\n
That’s it. Really. We don’t need anything else.<\/p>\n
Apache Camel has a Debezium component<\/a> that can hook up a MySQL database and use Debezium embedded engine<\/a>. The source endpoint configuration provides the parameters needed by Debezium to note any change happening in the debezium._order<\/code> table.\u00a0Debezium streams the events according to a JSON-defined format, so you know what kind of information to expect<\/a>. For each event, you will get the information as it was before<\/em> and after<\/em> the event occurs, plus a few useful pieces of meta-information.<\/p>\n
Thanks to Camel’s content-based router, we can either call the addOrderUsingPOST<\/code> or deleteOrderUsingDELETE<\/code> operation. You only have to develop a message translator that can convert the message coming from Debezium:<\/p>\n
public class AfterStructToOrderTranslator implements Processor { private static final String EXPECTED_BODY_FORMAT = \"{\\\"userId\\\":%d,\\\"orderId\\\":%d}\"; public void process(Exchange exchange) throws Exception { final Map value = exchange.getMessage().getBody(Map.class); \/\/ Convert and set body int userId = (int) value.get(\"user_id\"); int orderId = (int) value.get(\"order_id\"); exchange.getIn().setHeader(\"userId\", userId); exchange.getIn().setHeader(\"orderId\", orderId); exchange.getIn().setBody(String.format(EXPECTED_BODY_FORMAT, userId, orderId)); } } <\/pre>\n
Notice that we did not touch any of the base code for\u00a0Order<\/code> or User<\/code>. Now, turn off the Debezium process to simulate downtime. You will see that it can recover all events as soon as it turns back on!<\/p>\n
You can run the example provided by following the instructions on this GitHub repo<\/a>.<\/p>\n
Caveat<\/h2>\n
The example illustrated here uses Debezium’s embedded mode. For more consistent solutions, consider using the Kafka connect mode instead, or tuning the embedded engine<\/a> accordingly.<\/p>\n
<\/a><\/a><\/a><\/a><\/a><\/a><\/a>\"Share\"<\/a><\/p>\n
The post Decoupling microservices with Apache Camel and Debezium<\/a> appeared first on Red Hat Developer<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
The rise of microservices-oriented architecture brought us new development paradigms and mantras about independent development and decoupling. In such a scenario, we have to deal with a situation where we aim for independence, but we still need to react to state changes in different enterprise domains. I’ll use a simple and typical example in order […]<\/p>\n","protected":false},"author":2,"featured_media":107683,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[858,3],"tags":[865,869,866,859,867,868],"class_list":["post-107682","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-java-tutorials","category-tutorials","tag-apache-camel","tag-debezium","tag-integration","tag-java","tag-microservices","tag-red-hat-fuse"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/107682","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=107682"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/107682\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/107683"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=107682"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=107682"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=107682"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}