Data Engineer
📋 Role Overview & Responsibilities
About Our Client:Our healthcare client is committed to leveraging innovative data engineering solutions, particularly the Medallion architecture, to drive operational excellence and deliver transformative insights in patient care.
Must-Haves:- Minimum of 3 years' experience with Databricks, PySpark, SQL, Spark clusters, and Jupyter Notebooks.- Expertise in building data lakes using the Medallion architecture and working with delta tables in the delta file format.- Familiarity with CI/CD pipelines and Agile methodologies, ensuring efficient and collaborative development practices.- Strong understanding of ETL processes, data modeling, and data warehousing principles.- Experience with data visualization tools like Power BI is a plus.- Knowledge of cybersecurity data, particularly vulnerability scan data, is preferred.- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
Responsibilities:- Develop, test, and maintain large-scale data processing systems using Databricks, PySpark, SQL, Spark clusters, delta tables, and the innovative Medallion architecture.- Collaborate closely with cybersecurity analysts to gather data requirements and deliver effective solutions aligned with Medallion architecture principles.- Ensure data quality and implement robust data governance standards, leveraging the scalability and efficiency offered by the Medallion architecture.- Design and implement ETL processes, including data cleansing, transformation, and integration, optimizing performance within the delta file format framework.- Build and manage data lakes based on Medallion architecture principles, ensuring scalability, reliability, and adherence to best practices.- Monitor and optimize data pipelines, integrating CI/CD practices to streamline development and deployment processes.- Collaborate with cross-functional team members to implement data analytics projects, utilizing Jupyter Notebooks and other tools to harness the power of the Medallion architecture.- Embrace Agile methodologies throughout the development lifecycle to promote iterative and collaborative development practices, enhancing the effectiveness of Medallion-based solutions.
Frequently Asked Questions
How do I apply for the Data Engineer position at Gardner Resources Consulting, LLC? ▼
Click the “Apply for this Position” button on this page to submit your application directly to Gardner Resources Consulting, LLC without recruitment intermediary fees.
Is this position eligible for remote work or visa sponsorship? ▼
This position is located in Boston, MA with potential relocation and sponsorship assistance depending on candidate qualifications.
Are there any candidate fees on Hirely? ▼
No. Hirely is completely free for candidates. We never charge registration fees or placement fees.
Legitimate employers will never ask you to transfer funds, purchase equipment from unauthorized vendors, or pay application/visa fees. Hirely rigorously monitors listings, but always verify communication is from official corporate email domains.
Learn how to protect yourself against employment fraud →