Experienced Data Engineer with 6 Years of Expertise

Comprehensive Skill Set and Technologies

Programming Languages

Core Languages

Databases

Relational and NoSQL Databases

Big Data Technologies

Data Processing and Management

Frameworks

Web and Application Frameworks

Data Visualization

Visualization Tools

NLP Frameworks

Natural Language Processing

Libraries and Tools

Essential Libraries

Additional Tools

Utility and Networking Tools

Edge and Cloud Computing

Cloud Services and Platforms

Experience & Education

Basic Qualification and Certifications

Experience

Senior Data Engineer

Sep 2025 - present
bp

  • - Working on the design and development of a dynamic ingestion pipeline where configuration details are received through FastAPI and used to automatically provision and execute new data pipelines in Databricks.
  • - Responsible for storing and managing processed metadata in Databricks Catalog and ensuring seamless integration across Azure cloud infrastructure.
  • - Additionally, implemented workflow automation, monitoring, and notification mechanisms to support pipeline reliability, observability, and operational efficiency.

Senior Software Engineer

Sep 2023 - Sep 2025
Vibrent Health

  • - Led and scaled data engineering initiatives, building robust data pipelines using DBT, Dagster, Meltano, Snowflake, Python, and AWS to support analytics and business intelligence needs.
  • - Owned data orchestration and architecture, managing tool upgrades, schema evolution, and data modeling decisions, while contributing to performance and reliability improvements across the stack.
  • - Optimized cost and performance, reducing Snowflake usage costs via DuckDB-based in-memory computation and cutting query times by 50% through DBT model and table design optimization.
  • - Contributed to the open-source community by enhancing a PipelineWise extractor used in Meltano, improving ingestion reliability and integration capabilities.

Senior Engineer Cloud Services and Software

March 2022 to Sep 2023
Larsen & Toubro Infotech Limited

  • - As a Senior Data Engineer, engaged in designing, implementing and optimizing the data pipeline along with Data Migration.
  • - While Working as a module lead, managing a team of 4-6 data engineers and analysts
  • - Utilized AWS cloud Platform services like Athena, Glue, EMR, SageMaker, Cloud-formation, S3, Lambda, DynamoDB, IAM, AWS-SAM, SNS, API Gateway
  • - Involved in Code development and peer reviewing using Python, Pyspark, Pandas, Docker and SQL
  • - Writing unit test cases using Pytest
  • - Creating report for products and collaborating with cross functional teams
  • - Involved in creating documentation for data pipeline architecture and Standard Operating Procedure
  • - Implemented data transformation processes, improving the quality of data
  • - Optimized and tuned database performance to achieve faster query response times and reduced latency.
  • - Created SDK for the user-event–logger ( iife, typescript, lambda, s3, cdn, npm package)
  • - MongoDb for user-event storage MongoDb
  • - Kafka(AWS MSK) for user event record

Software Engineer

June 2020 to March 2022
Clearmind Consultancy

  • - As a Data Engineer, Managing the transaction ledger and data transformation as per business need. Writing Stock market related scripts(ticker) and Analysis. Designing Cloud Architect for data pipeline, ETL , web app, server and data migration
  • - Data Encryption and decryption of the transaction ledger( Python, DynamoDb)
  • - Kafka for the Share price update
  • - Auto invoice and billing generation
  • - Razorpay integration
  • - Designing algorithm for StockMarket Price Analysis based on ticker values using Pine Script.
  • - Using CCXT created serverless portfolio handler (with AWS-SAM)
  • - Created sdk for aadhar offline ekyc(Lambda, S3, API gateway, Javascript, AWS-SAM)
  • - React SSR
  • - Developed, designed and productionize Android & IOS App and WebApp.
  • - Developed the server based on Node Js, mongo, restApi, Mysql, firebase
  • - Designed database for the cycle app and created graphql api based access
  • - Pod deployment of webApp and database pipeline on aws
  • - Created reverse proxy using nginx for cycle server and cycle webapp
  • - Created an api for scrapping of nse bse data, feed it to the analysis of ticker

Associate Engineer

June 2018 to August 2019
Solabot Technologies

  • - As an Engineer, Writing automation scripts for sensor data. Analysis of sensor data based on bot activity. Creating the layout and design of the Bots
  • - Bot real time location, working status Tracking
  • - AWS lambda for API building
  • - Generated the crash analytics report Used python for the report analysis.
  • - Static webPage hosted in S3
  • - Used Python, Embedded C for Arduino Programming.
  • - Created layout design for solar panel cleaning bot using SolidWorks, AutoCad.
  • - Writing automation scripts for reducing manual and redundant tasks.

Education

Post Graduation Diploma in Internet of Things

2019 - 2020
Center for Development of Advanced Computing, Pune

    Bachelor of Technology, Mechatronics

    2014 - 2018
    Kurukshetra University

      Paper Published

      Quantum Computing for Data Integration and Optimization

      ISSN: 2393-9028 (PRINT) | ISSN: 2348-2281 (ONLINE)
      IJRECE VOL. 13 ISSUE 2 APR-JUNE 2025

      • - This research explores how quantum algorithms like QFT and QAOA can transform ETL workflows.
      • - We demonstrate quantum-based data compression and intelligent task scheduling using Qiskit and Airflow.
      • - This approach improves speed, scalability, and efficiency in handling massive datasets.
      • - Published in IJRECE, Vol. 13, Issue 2 (2025).
      • - To view paper

      Made by Devendra Kumar Singh