Lead Data Engineer Job at Reuben Cooley Inc., Cary, NC

  • Reuben Cooley Inc.
  • Cary, NC

Job Description

Programming & Data Engineering

  • Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.
  • Strong SQL and data modelling dimensional and normalised; schema design and data contract definition.
  • Databricks expertise Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
  • Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.

AI & Machine Learning

  • 3+ years designing and shipping LLM-based systems in production: RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
  • Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.
  • Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
  • Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.

Architecture & Governance

  • 12–18 years of total experience in data engineering, data platform delivery, or related disciplines.
  • Proven delivery of a medallion/lakehouse architecture at enterprise scale — not just familiarity with the concept.
  • Azure security and governance: Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
  • CI/CD and infrastructure as code: Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
  • Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.

Job Tags

Full time, Contract work

Similar Jobs

Ping Identity

Director, Global Payroll Job at Ping Identity

 ...experiences, and identity and access management. We are seeking an experienced, strategic, and highly motivated Director, Global Payroll to lead our payroll team. Reporting to the SVP Controller, this leader will oversee payroll operations across 18 countries,... 

Swedish Health Services

Senior Ultrasound Technologist - Maternal Fetal Medicine Job at Swedish Health Services

 ...professionally and achieving financial security. We take care of you, so you can focus on delivering our mission of improving the health and wellbeing of each patient we serve. About Providence At Providence, our strength lies in Our Promise of Know me, care... 

All American Home Care LLC

*** $15 hr Flyer Distributor *** Job at All American Home Care LLC

 ...being promoted. Maintain a positive and professional image while representing the company in public spaces. Follow designated routes and schedules for efficient distribution. Report back on the number of distributed materials and gather feedback from the public... 

NRS

Driver-Semi-Regional Job at NRS

 ...skill to the challenges facing our clients. In the office, the warehouse, or on the road, it is this commitment to our innovative...  ...Schedule: Monday to Friday Home Time: Mid-week, off weekends Service Area: NE & NY Routes Paid Orientation & Safety... 

Christus Health

CRNA - Certified Registered Nurse Anesthetist Job at Christus Health

 ...Description Join one of Texas largest and most respected multispecialty groupsCHRISTUS Trinity Clinicas we seek a BC/BE CRNA to become part of our mission-driven, faith-based organization in Mt. Pleasant, TX. This is a unique opportunity to practice in aPhysician...