Which data synchronization and analysis solution to choose (ESB, EAI or other)?

Question

data mining

My customer's partners store data in different databases or files ( postgresql , oracle , mssql , mysql , dbf , csv ), or access the data via API. I need to get the data, convert it, send it to the customer's database and analyze it further.
If directly, the solution seems to be the following: using the replication protocol or triggers, we get data from databases for SQL, for files, you can use something like diff. Then the data is converted into a form that is stored by the customer and uploaded to the customer's database. Parsing is done with pandas , apach pig or whatever.
I found information that there are all kinds of Enterprise Application Integration and Enterprise Service Bus , there are also solutions for replicating different types of databases like SymmetricDS .
After looking a little at Talend , Mule ESB , Fuse ESB , Apache ServiceMix , Zato , I realized that the threshold for entering them is quite large.
After thinking, I made up the requirements for the system:

1. Data synchronization
   
   • Synchronization of transactional SQL databases using replication or triggers
     
   • File handling and API
     
   • Storing a copy of the data in the system for analysis
     
2. Data integrity
   
   • Referential Integrity
     
   • transactional
     
3. Data conversion
   
   • Easy addition of new partners
     
   • Setting up the conversion of partner data to customer data
     
   • Apply changes on the fly
     
4. Service health monitoring
   
5. Expandability
   
6. Security
   
   • Using Encryption
     
   • Easy addition of a node on the partner side for proxying when there is no direct access to the database
     
7. Data analysis with pandas , apache pig or other
   

Perhaps someone will tell you the simplest solution that covers my requirements as much as possible, or can someone tell / suggest how, with the help of any particular tool, you can solve / facilitate the solution of the problem?

Answer 1

I wonder if the issue has been resolved. If yes, please describe the solution.
For my scenario of integrating several entities from Bitrix24 and entities in the customer's own ERP system, I am also looking for the simplest solution. I did not find such solutions in RuNet.
Here is a selected list of foreign servers:
help.mydbsync.com/docs/display/dbsync/Home
www.jitterbit.com/integrate
www.informaticacloud.com/products/cloud-services/d...
www.onesaas.com
www.skyscapecloud .com/software-as-a-service/sync-a...
But I come to the conclusion that I need to develop it myself. Perhaps we should cooperate and solve our problems by joint development.