patternsqlMinor
Does MySQL have a version of Change Data Capture?
Viewed 0 times
versioncapturemysqldoesdatachangehave
Problem
We're in the process of phasing out an old system and migrating onto a new one. The last time that we phased out an old system, we ran both systems in parallel and integrated data between both until everything was fully migrated out in our field. During that process, I was able to build an integration between our legacy system and our new system that leveraged SQL Server's Change Data Capture to track changes and integrate those over incrementally.
For this next migration, the legacy system that we will be phasing out is based on MySQL v5.1.69 instead of SQL Server. I am not familiar with MySQL and I was wondering if there are any technologies similar to CDC that can be leveraged on MySQL in our current version or a newer version that would be worth migrating towards?
For this next migration, the legacy system that we will be phasing out is based on MySQL v5.1.69 instead of SQL Server. I am not familiar with MySQL and I was wondering if there are any technologies similar to CDC that can be leveraged on MySQL in our current version or a newer version that would be worth migrating towards?
Solution
LinkedIn has open-sourced its CDC tool, Databus, which supports MySQL. You can read about how to support MySQL on the wiki of the project: Databus for MySQL and read the blog statement here: Open sourcing Databus: LinkedIn's low latency change data capture system
However, you should note that there may be some limitations:
The current implementation although fully functional should be considered a proof-of-concept. Its main goal is to demonstrate how to write a MySQL fetcher for Databus. The basic functionality of the fetcher has been tested but has not been deployed in our production environment. At LinkedIn, we run a slightly different implementation which relies on a custom-patched MySQL.
Another open-source CDC tool is FlexCDC (implemented in PHP) which was written as a part of the FlexViews project.
[...] FlexCDC which will aim to provide a flexible CDC layer for MySQL based on RBR logging, and Flexviews, which uses FlexCDC to collect the changes necessary to incrementally maintain materialized views.
From: Why are there no (other) change data capture utilities for MySQL 5.1?
However, you should note that there may be some limitations:
The current implementation although fully functional should be considered a proof-of-concept. Its main goal is to demonstrate how to write a MySQL fetcher for Databus. The basic functionality of the fetcher has been tested but has not been deployed in our production environment. At LinkedIn, we run a slightly different implementation which relies on a custom-patched MySQL.
Another open-source CDC tool is FlexCDC (implemented in PHP) which was written as a part of the FlexViews project.
[...] FlexCDC which will aim to provide a flexible CDC layer for MySQL based on RBR logging, and Flexviews, which uses FlexCDC to collect the changes necessary to incrementally maintain materialized views.
From: Why are there no (other) change data capture utilities for MySQL 5.1?
Context
StackExchange Database Administrators Q#63681, answer score: 6
Revisions (0)
No revisions yet.