Job DescriptionLead Data Engineer
As a Lead Data Engineer, you’ll apply strong expertise in Artificial Intelligence through the use of data engineering, ML, data mining, & data retrieval to design, prototype, build & deploy next gen advanced analytics engines & services. You will collaborate with cross-functional teams & business partners to define the technical problem statement & hypotheses to test. You will have the opportunity to drive current & future strategy by leveraging your analytical skills as you ensure business value & communicate the results.
10+ years of experience in implementing & managing high performance scalable enterprise applications in the Financial Services industry.
Extensive experience on the Hadoop platform & related Big Data tools/technologies.
Good knowledge of architecture, design patterns, Source target mappings, ETL Architecture in Hadoop space, data modeling techniques, performance tuning in Hadoop environment.
Responsible for helping team turn data into knowledge to help them make better decision, faster.
Work with clients & team members to analyze & help define requirements, mine & analyze data, integrate data from a variety of sources, & participate in the design & implementation of reports, algorithms, & other data processing & analysis techniques.
Deliver high-quality data pipelines for producing analytics-ready datasets.
Responsible for delivering end-to-end analytics projects including, data: ingest, transportation, silence & visualization.
Design & deploy databases(Tenants) & data pipelines to support analytics projects.
Clearly document datasets, solutions, findings, & recommendations to be shared internally & externally.
Apply technologies proficiently including: SQL/Hive, Python, PySpark, Spark/Mapreduce, Bash, Hadoop, Azure, Oozie WF, Data science Anaconda Enterprise, Jupyter, Tableau.
Complete performance optimization for queries & dashboards, & develop & deliver clear, compelling briefings to internal & external stakeholders on his findings, recommendations & solutions.
Analyze client data & systems to determine whether requirements can be met.
Test & validate data pipelines transformations, datasets, reports & dashboards built by the team.
Develop & communicate solutions architectures & present solutions to both business & technical stakeholders.
Provide end user support to other data engineers & analysts.
Build compelling visualizations & dashboards that address the analytics needs of the end-user or customer.