01-21-2020, 12:50 AM
Decoupling microservices with Apache Camel and Debezium
<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/01/decoupling-microservices-with-apache-camel-and-debezium.png" width="618" height="331" title="" alt="" /></div><div><p>The rise of <a href="https://developers.redhat.com/topics/microservices/">microservices</a>-oriented architecture brought us new development paradigms and mantras about independent development and decoupling. In such a scenario, we have to deal with a situation where we aim for independence, but we still need to react to state changes in different enterprise domains.</p>
<p>I’ll use a simple and typical example in order to show what we’re talking about. Imagine the development of two independent microservices: <code>Order</code> and <code>User</code>. We designed them to expose a REST interface and to each use a separate database, as shown in Figure 1:</p>
<div id="attachment_648347" style="width: 628px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648347" class="wp-image-648347 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-1.png" alt="Diagram 1 - Order and User microservices" width="618" height="331" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-1.png 618w, https://developers.redhat.com/blog/wp-co...00x161.png 300w" sizes="(max-width: 618px) 100vw, 618px" /></p>
<p id="caption-attachment-648347" class="wp-caption-text">Figure 1: Order and User microservices.</p>
</div>
<p><span id="more-648217"></span></p>
<p>We must notify the <code>User</code> domain about any change happening in the <code>Order</code> domain. To do this in the example, we need to update the <code>order_list</code>. For this reason, we’ve modeled the User REST service with <code>addOrder</code> and <code>deleteOrder</code> operations.</p>
<h2>Solution 1: Queue decoupling</h2>
<p>The first solution to consider is adding a queue between the services. <code>Order</code> will publish events that <code>User</code> will eventually process, as shown in Figure 2:</p>
<div id="attachment_648337" style="width: 338px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648337" class="wp-image-648337 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-2.png" alt="Diagram 2 - decoupling with a queue" width="328" height="431" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-2.png 328w, https://developers.redhat.com/blog/wp-co...28x300.png 228w" sizes="(max-width: 328px) 100vw, 328px" /></p>
<p id="caption-attachment-648337" class="wp-caption-text">Figure 2: Decoupling with a queue.</p>
</div>
<p>This is a fair design. However, if you don’t use the right middleware you will mix a lot of <em>infrastructure code</em> into your <em>domain logic. </em>Now that you have queues, you must develop <em>producer</em> and <em>consumer logic</em>. You also have to take care of <em>transactions. </em>The problem is to make sure that every event ends up correctly in both the <code>Order</code> database and in the queue.</p>
<h2>Solution 2: Change data capture decoupling</h2>
<p>Let me introduce an alternative solution that handles all of that work without your touching any line of your microservices code. I’ll use <a href="https://debezium.io/" target="_blank" rel="noopener noreferrer">Debezium</a> and <a href="https://camel.apache.org/" target="_blank" rel="noopener noreferrer">Apache Camel</a> to capture data changes on <code>Order</code> and trigger certain actions on <code>User</code>. Debezium is a log-based data change capture middleware. Camel is an integration framework that simplifies the integration between a source (<code>Order</code>) and a destination (<code>User</code>), as shown in Figure 3:</p>
<div id="attachment_648327" style="width: 501px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648327" class="wp-image-648327 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-3.png" alt="Diagram 3 - decoupling with Debezium and Camel" width="491" height="429" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-3.png 491w, https://developers.redhat.com/blog/wp-co...00x262.png 300w" sizes="(max-width: 491px) 100vw, 491px" /></p>
<p id="caption-attachment-648327" class="wp-caption-text">Figure 3: Decoupling with Debezium and Camel.</p>
</div>
<p>Debezium is in charge of capturing any data change happening in the <code>Order</code> domain and publishing it to a <em>topic</em>. Then a Camel consumer can pick that event and make a REST call to the <code>User</code> API to perform the necessary action expected by its domain (in our simple case, update the list).</p>
<h2>Decoupling with Debezium and Camel</h2>
<p>I’ve prepared a simple demo with all of the components we need to run the example above. You can find this demo <a href="https://github.com/squakez/debezium-camel-demo" target="_blank" rel="noopener noreferrer">in this GitHub repo</a>. The only part we need to develop is represented by the following source code:</p>
<pre>public class MyRouteBuilder extends RouteBuilder { public void configure() { from("debezium:mysql?name=my-sql-connector" + "&databaseServerId=1" + "&databaseHostName=localhost" + "&databaseUser=debezium" + "&databasePassword=dbz" + "&databaseServerName=my-app-connector" + "&databaseHistoryFileName=/tmp/dbhistory.dat" + "&databaseWhitelist=debezium" + "&tableWhitelist=debezium._order" + "&offsetStorageFileName=/tmp/offset.dat") .choice() .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo("c")) .process(new AfterStructToOrderTranslator()) .to("rest-swagger:http://localhost:8082/v2/api-docs#addOrderUsingPOST") .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo("d")) .process(new BeforeStructToOrderTranslator()) .to("rest-swagger:http://localhost:8082/v2/api-docs#deleteOrderUsingDELETE") .log("Response : ${body}"); } } </pre>
<p>That’s it. Really. We don’t need anything else.</p>
<p>Apache Camel has a <a href="https://camel.apache.org/components/latest/debezium-mysql-component.html">Debezium component</a> that can hook up a MySQL database and use <a href="https://debezium.io/documentation/reference/0.9/operations/embedded.html" target="_blank" rel="noopener noreferrer">Debezium embedded engine</a>. The source endpoint configuration provides the parameters needed by Debezium to note any change happening in the <code>debezium._order</code> table. Debezium streams the events according to a JSON-defined format, so you know what kind of <a href="https://debezium.io/documentation/reference/0.10/connectors/mysql.html#events" target="_blank" rel="noopener noreferrer">information to expect</a>. For each event, you will get the information as it was <em>before</em> and <em>after</em> the event occurs, plus a few useful pieces of meta-information.</p>
<p>Thanks to Camel’s content-based router, we can either call the <code>addOrderUsingPOST</code> or <code>deleteOrderUsingDELETE</code> operation. You only have to develop a message translator that can convert the message coming from Debezium:</p>
<pre>public class AfterStructToOrderTranslator implements Processor { private static final String EXPECTED_BODY_FORMAT = "{\"userId\":%d,\"orderId\":%d}"; public void process(Exchange exchange) throws Exception { final Map value = exchange.getMessage().getBody(Map.class); // Convert and set body int userId = (int) value.get("user_id"); int orderId = (int) value.get("order_id"); exchange.getIn().setHeader("userId", userId); exchange.getIn().setHeader("orderId", orderId); exchange.getIn().setBody(String.format(EXPECTED_BODY_FORMAT, userId, orderId)); } } </pre>
<p>Notice that we did not touch any of the base code for <code>Order</code> or <code>User</code>. Now, turn off the Debezium process to simulate downtime. You will see that it can recover all events as soon as it turns back on!</p>
<p>You can run the example provided by<a href="https://github.com/squakez/debezium-camel-demo" target="_blank" rel="noopener noreferrer"> following the instructions on this GitHub repo</a>.</p>
<h2>Caveat</h2>
<p>The example illustrated here uses Debezium’s embedded mode. For more consistent solutions, consider using the Kafka connect mode instead, or <a href="https://debezium.io/documentation/reference/0.10/operations/embedded.html#_handling_failures" target="_blank" rel="noopener noreferrer">tuning the embedded engine</a> accordingly.</p>
<p><a class="a2a_button_facebook" href="https://www.addtoany.com/add_to/facebook?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Facebook" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_twitter" href="https://www.addtoany.com/add_to/twitter?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Twitter" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_linkedin" href="https://www.addtoany.com/add_to/linkedin?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="LinkedIn" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_email" href="https://www.addtoany.com/add_to/email?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Email" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_print" href="https://www.addtoany.com/add_to/print?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Print" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_reddit" href="https://www.addtoany.com/add_to/reddit?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Reddit" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_flipboard" href="https://www.addtoany.com/add_to/flipboard?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Flipboard" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_dd addtoany_share_save addtoany_share" href="https://www.addtoany.com/share#url=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&title=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" data-a2a-url="https://developers.redhat.com/blog/2019/11/19/decoupling-microservices-with-apache-camel-and-debezium/" data-a2a-title="Decoupling microservices with Apache Camel and Debezium"><img src="https://static.addtoany.com/buttons/favicon.png" alt="Share"></a></p>
<p>The post <a rel="nofollow" href="https://developers.redhat.com/blog/2019/11/19/decoupling-microservices-with-apache-camel-and-debezium/">Decoupling microservices with Apache Camel and Debezium</a> appeared first on <a rel="nofollow" href="https://developers.redhat.com/blog">Red Hat Developer</a>.</p>
</div>
https://www.sickgaming.net/blog/2019/11/...-debezium/
<div style="margin: 5px 5% 10px 5%;"><img src="https://www.sickgaming.net/blog/wp-content/uploads/2020/01/decoupling-microservices-with-apache-camel-and-debezium.png" width="618" height="331" title="" alt="" /></div><div><p>The rise of <a href="https://developers.redhat.com/topics/microservices/">microservices</a>-oriented architecture brought us new development paradigms and mantras about independent development and decoupling. In such a scenario, we have to deal with a situation where we aim for independence, but we still need to react to state changes in different enterprise domains.</p>
<p>I’ll use a simple and typical example in order to show what we’re talking about. Imagine the development of two independent microservices: <code>Order</code> and <code>User</code>. We designed them to expose a REST interface and to each use a separate database, as shown in Figure 1:</p>
<div id="attachment_648347" style="width: 628px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648347" class="wp-image-648347 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-1.png" alt="Diagram 1 - Order and User microservices" width="618" height="331" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-1.png 618w, https://developers.redhat.com/blog/wp-co...00x161.png 300w" sizes="(max-width: 618px) 100vw, 618px" /></p>
<p id="caption-attachment-648347" class="wp-caption-text">Figure 1: Order and User microservices.</p>
</div>
<p><span id="more-648217"></span></p>
<p>We must notify the <code>User</code> domain about any change happening in the <code>Order</code> domain. To do this in the example, we need to update the <code>order_list</code>. For this reason, we’ve modeled the User REST service with <code>addOrder</code> and <code>deleteOrder</code> operations.</p>
<h2>Solution 1: Queue decoupling</h2>
<p>The first solution to consider is adding a queue between the services. <code>Order</code> will publish events that <code>User</code> will eventually process, as shown in Figure 2:</p>
<div id="attachment_648337" style="width: 338px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648337" class="wp-image-648337 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-2.png" alt="Diagram 2 - decoupling with a queue" width="328" height="431" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-2.png 328w, https://developers.redhat.com/blog/wp-co...28x300.png 228w" sizes="(max-width: 328px) 100vw, 328px" /></p>
<p id="caption-attachment-648337" class="wp-caption-text">Figure 2: Decoupling with a queue.</p>
</div>
<p>This is a fair design. However, if you don’t use the right middleware you will mix a lot of <em>infrastructure code</em> into your <em>domain logic. </em>Now that you have queues, you must develop <em>producer</em> and <em>consumer logic</em>. You also have to take care of <em>transactions. </em>The problem is to make sure that every event ends up correctly in both the <code>Order</code> database and in the queue.</p>
<h2>Solution 2: Change data capture decoupling</h2>
<p>Let me introduce an alternative solution that handles all of that work without your touching any line of your microservices code. I’ll use <a href="https://debezium.io/" target="_blank" rel="noopener noreferrer">Debezium</a> and <a href="https://camel.apache.org/" target="_blank" rel="noopener noreferrer">Apache Camel</a> to capture data changes on <code>Order</code> and trigger certain actions on <code>User</code>. Debezium is a log-based data change capture middleware. Camel is an integration framework that simplifies the integration between a source (<code>Order</code>) and a destination (<code>User</code>), as shown in Figure 3:</p>
<div id="attachment_648327" style="width: 501px" class="wp-caption aligncenter"><img aria-describedby="caption-attachment-648327" class="wp-image-648327 size-full" src="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-3.png" alt="Diagram 3 - decoupling with Debezium and Camel" width="491" height="429" srcset="https://developers.redhat.com/blog/wp-content/uploads/2019/11/diagram-3.png 491w, https://developers.redhat.com/blog/wp-co...00x262.png 300w" sizes="(max-width: 491px) 100vw, 491px" /></p>
<p id="caption-attachment-648327" class="wp-caption-text">Figure 3: Decoupling with Debezium and Camel.</p>
</div>
<p>Debezium is in charge of capturing any data change happening in the <code>Order</code> domain and publishing it to a <em>topic</em>. Then a Camel consumer can pick that event and make a REST call to the <code>User</code> API to perform the necessary action expected by its domain (in our simple case, update the list).</p>
<h2>Decoupling with Debezium and Camel</h2>
<p>I’ve prepared a simple demo with all of the components we need to run the example above. You can find this demo <a href="https://github.com/squakez/debezium-camel-demo" target="_blank" rel="noopener noreferrer">in this GitHub repo</a>. The only part we need to develop is represented by the following source code:</p>
<pre>public class MyRouteBuilder extends RouteBuilder { public void configure() { from("debezium:mysql?name=my-sql-connector" + "&databaseServerId=1" + "&databaseHostName=localhost" + "&databaseUser=debezium" + "&databasePassword=dbz" + "&databaseServerName=my-app-connector" + "&databaseHistoryFileName=/tmp/dbhistory.dat" + "&databaseWhitelist=debezium" + "&tableWhitelist=debezium._order" + "&offsetStorageFileName=/tmp/offset.dat") .choice() .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo("c")) .process(new AfterStructToOrderTranslator()) .to("rest-swagger:http://localhost:8082/v2/api-docs#addOrderUsingPOST") .when(header(DebeziumConstants.HEADER_OPERATION).isEqualTo("d")) .process(new BeforeStructToOrderTranslator()) .to("rest-swagger:http://localhost:8082/v2/api-docs#deleteOrderUsingDELETE") .log("Response : ${body}"); } } </pre>
<p>That’s it. Really. We don’t need anything else.</p>
<p>Apache Camel has a <a href="https://camel.apache.org/components/latest/debezium-mysql-component.html">Debezium component</a> that can hook up a MySQL database and use <a href="https://debezium.io/documentation/reference/0.9/operations/embedded.html" target="_blank" rel="noopener noreferrer">Debezium embedded engine</a>. The source endpoint configuration provides the parameters needed by Debezium to note any change happening in the <code>debezium._order</code> table. Debezium streams the events according to a JSON-defined format, so you know what kind of <a href="https://debezium.io/documentation/reference/0.10/connectors/mysql.html#events" target="_blank" rel="noopener noreferrer">information to expect</a>. For each event, you will get the information as it was <em>before</em> and <em>after</em> the event occurs, plus a few useful pieces of meta-information.</p>
<p>Thanks to Camel’s content-based router, we can either call the <code>addOrderUsingPOST</code> or <code>deleteOrderUsingDELETE</code> operation. You only have to develop a message translator that can convert the message coming from Debezium:</p>
<pre>public class AfterStructToOrderTranslator implements Processor { private static final String EXPECTED_BODY_FORMAT = "{\"userId\":%d,\"orderId\":%d}"; public void process(Exchange exchange) throws Exception { final Map value = exchange.getMessage().getBody(Map.class); // Convert and set body int userId = (int) value.get("user_id"); int orderId = (int) value.get("order_id"); exchange.getIn().setHeader("userId", userId); exchange.getIn().setHeader("orderId", orderId); exchange.getIn().setBody(String.format(EXPECTED_BODY_FORMAT, userId, orderId)); } } </pre>
<p>Notice that we did not touch any of the base code for <code>Order</code> or <code>User</code>. Now, turn off the Debezium process to simulate downtime. You will see that it can recover all events as soon as it turns back on!</p>
<p>You can run the example provided by<a href="https://github.com/squakez/debezium-camel-demo" target="_blank" rel="noopener noreferrer"> following the instructions on this GitHub repo</a>.</p>
<h2>Caveat</h2>
<p>The example illustrated here uses Debezium’s embedded mode. For more consistent solutions, consider using the Kafka connect mode instead, or <a href="https://debezium.io/documentation/reference/0.10/operations/embedded.html#_handling_failures" target="_blank" rel="noopener noreferrer">tuning the embedded engine</a> accordingly.</p>
<p><a class="a2a_button_facebook" href="https://www.addtoany.com/add_to/facebook?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Facebook" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_twitter" href="https://www.addtoany.com/add_to/twitter?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Twitter" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_linkedin" href="https://www.addtoany.com/add_to/linkedin?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="LinkedIn" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_email" href="https://www.addtoany.com/add_to/email?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Email" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_print" href="https://www.addtoany.com/add_to/print?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Print" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_reddit" href="https://www.addtoany.com/add_to/reddit?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Reddit" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_button_flipboard" href="https://www.addtoany.com/add_to/flipboard?linkurl=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&linkname=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" title="Flipboard" rel="nofollow noopener noreferrer" target="_blank"></a><a class="a2a_dd addtoany_share_save addtoany_share" href="https://www.addtoany.com/share#url=https%3A%2F%2Fdevelopers.redhat.com%2Fblog%2F2019%2F11%2F19%2Fdecoupling-microservices-with-apache-camel-and-debezium%2F&title=Decoupling%20microservices%20with%20Apache%20Camel%20and%20Debezium" data-a2a-url="https://developers.redhat.com/blog/2019/11/19/decoupling-microservices-with-apache-camel-and-debezium/" data-a2a-title="Decoupling microservices with Apache Camel and Debezium"><img src="https://static.addtoany.com/buttons/favicon.png" alt="Share"></a></p>
<p>The post <a rel="nofollow" href="https://developers.redhat.com/blog/2019/11/19/decoupling-microservices-with-apache-camel-and-debezium/">Decoupling microservices with Apache Camel and Debezium</a> appeared first on <a rel="nofollow" href="https://developers.redhat.com/blog">Red Hat Developer</a>.</p>
</div>
https://www.sickgaming.net/blog/2019/11/...-debezium/