Implementing CI/CD in Databricks Using Databricks Asset BundlesIn this article, we’ll explore a different CI/CD approach for deploying notebooks and scripts in Databricks using Databricks Asset Bundles…3d ago3d ago
Implementing CI/CD in Databricks Using Repos APII recently started working on a new project that uses Databricks as its data platform. While I have worked with Databricks in the past, the…Mar 28Mar 28
Implementing CI/CD for dbt-core with BigQuery and Github ActionsIn this article, I’ll walk you through setting up a CI/CD pipeline for dbt-core using BigQuery and GitHub Actions. The goal is to show you…Mar 42Mar 42
Real-Time End-to-End Data Streaming with Postgres, Kafka, MinIO, Prometheus, and GrafanaAs data engineering evolves, proficiency in both batch and streaming architectures is increasingly essential. Batch processing handles…Jul 9, 2024Jul 9, 2024
CI/CD and DevOps for Snowflake: A Comprehensive GuideIn the world of data engineering and database management, seamlessly integrating Continuous Integration/Continuous Deployment (CI/CD)…Jun 13, 20243Jun 13, 20243
10 Things I Like About Snowflake — from a SQL Server MigrantAs a data engineer, I’ve spent much of my career working with SQL Server, managing data pipelines and extracting insights within its…May 9, 20241May 9, 20241
Metadata-Driven Development in Azure Data FactoryIn the realm of data engineering, harnessing the power of metadata-driven development is not merely a revolutionary concept — it’s a…Mar 24, 2024Mar 24, 2024
CDC 101 in SQL Server for Data EngineersWhile Change Data Capture (CDC) is a concept not limited to SQL Server, this article will focus on its implementation within the SQL Server…Mar 12, 2024Mar 12, 2024
Displaying Self-Hosted Agents on Classic Release Pipeline (Azure DevOps)Recently, in my role as a data engineer, I undertook the task of converting our Azure SQL database deployment pipeline from YAML to the…Mar 1, 2024Mar 1, 2024
Deploying Python Applications for Free using RenderDeploying Python applications can be a daunting task, especially if you’re on a tight budget. While Heroku is a popular choice for many…Sep 22, 20231Sep 22, 20231
