Dice is the leading career destination for tech experts at every stage of their careers. Our client, ERPA, is seeking the following. Apply via Dice today!
Must Have Technologies:
• Azure
• Airflow
• Spark
• Hive
• Kubernetes
BDP Platform Engineer — Contractor Lead (Onsite)
Client: US Bank
Locations: Minneapolis, Atlanta, or San Francisco
Duration: 12-24 months
Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite–offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.
Key Responsibilities
• Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
• ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
• Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
• Onsite–offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
• CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
• Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.
Requirements
• Experience: 4–8+ years in platform/data engineering with production ownership.
• Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
• Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
• Languages: Proficient in Python, Scala, and Java.
• DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
• Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.
Tech Stack Highlights
• Azure (ADLS, AKS, ACR, Event Hubs, Key Vault, Monitor), Databricks/Synapse
• Apache Airflow, Apache Spark, Hive; optional Delta Lake/Iceberg
• GitLab Actions/Azure DevOps, Terraform/Ansible, Docker/Kubernetes
• Datadog
Thanks & Regards,
Anthony Kumar Guvvala
ERP Analysts, Inc.
Direct:
Email :
Apply Now
Apply Now