Job Description

Data Engineer 

Responsibilities:

• Designing and implementing fine-tuned production ready data/ML pipelines in Hadoop platform.
• Driving optimization, testing and tooling to improve quality.
• Reviewing and approving high level & amp; detailed design to ensure that the solution delivers to the business needs and aligns to the data & analytics architecture principles and roadmap.
• Understanding business requirements and solution design to develop and implement solutions that adhere to big data architectural guidelines and address business requirements.
• Following proper SDLC (Code review, sprint process).
• Identifying, designing, and implementing internal process improvements: automating manual processes, optimizing data delivery, etc.
• Building robust and scalable data infrastructure (both batch processing and real-time) to support needs from internal and external users.
• Understanding various data security standards and using secure data security tools to apply and adhere to the required data controls for user access in the Hadoop platform.
• Supporting and contributing to development guidelines and standards for data ingestion.
• Working with a data scientist and business analytics team to assist in data ingestion and data related technical issues.
• Designing and documenting the development & deployment flow.

Requirements:

• Experience in developing rest API services using one of the Scala frameworks.
• Ability to troubleshoot and optimize complex queries on the Spark platform
• Expert in building and optimizing ‘big data’ data/ML pipelines, architectures and data sets.
• Knowledge in modelling unstructured to structured data design.
• Experience in Big Data access and storage techniques.
• Experience in doing cost estimation based on the design and development.
• Excellent debugging skills for the technical stack mentioned above which even includes analyzing server logs and application logs.
• Highly organized, self-motivated, proactive, and ability to propose best design solutions.
• Good time management and multitasking skills to work to deadlines by working independently and as a part of a team.