An important task in Data Engineering is writing efficient scalable pipelines that move data from A to B. There are many GUI-tools out there and they are great because there is a real shortage of Data Engineers at the moment.
Being Data Engineers however we should not be limited by what any tool can do for us, locked in by any vendor or having to spent time figuring out how a brandnew hip GUI works every few months.
Therefore it is preferable to build a toolbox with migration-scripts for the most common use-cases and this series of articles is intended…
Migration a MySQL Database to GCP Firebase using Python and Pandas
I have a little VM on a well-known Developer Cloud which I love for quick prototyping for new and prospective clients.
It has all my Data Science and Machine Learning Libraries installed, a Flask Webserver, built on uWSGI behind an Nginx Reverse Proxy, for storage I have installed and connected MongoDB, Redis and MySQL.
Therefore, it is easy for me to quickly scrape some data or train an ML model and put it in a somewhat presentable form onto this server to demonstrate what I have built.