Key Responsibilities:

• Design, develop, and maintain scalable and high-performance data pipelines using tools such as PySpark, SQL, Databricks, and ADF.
• Perform data integration, transformation, and optimization tasks to ensure data is ready for analysis and reporting.
• Collaborate with data scientists and business analysts to define data requirements and implement solutions.
• Implement and maintain robust data modeling practices that ensure optimal data structure, integrity, and security.
• Automate data workflows, implement CI/CD processes, and manage version control for data pipeline processes using Azure DevOps.
• Leverage Azure Data Lake Storage (ADLS) for the storage of structured and unstructured data, optimizing data storage and retrieval.
• Implement best practices for data governance, ensuring the security, privacy, and quality of the data pipeline.
• Troubleshoot and optimize data pipelines for performance, scalability, and reliability.
• Provide technical documentation and training for teams on data infrastructure and pipeline usage.
• Work closely with stakeholders to ensure that data is accessible, clean, and accurate for reporting and analytic