Python Job: Web Developer

Job added on


Pune - India

Job type


Python Job Details

Job description

  • Proven experience as Web Scrapper/Crawler or similar role 2-4 years’ experience with a Bachelor's Degree in Computer Science, Engineering, Technology or related field required
  • Have strong understanding and working knowledge of web crawlers, web scrapers and other automation tools, to help browse the web content
  • Knowledge of web scraping and tools
  • Strong knowledge of any of multiple open-source and proprietary scraping frameworks available
  • Hands-on-experience with SQL/NO-SQL (MySQL/ Postgres/Cassandra /MongoDB)
  • Good knowledge and coding experience in one or more programming languages such as Python, Java, JavaScript
  • Experience of creating scrapy spiders for websites with Captcha, IP ban, geolocation ban, Cloudflare / Distil / Imperva firewalls, sites required login to access data, Dynamic websites loading through JS / REST API / Graphql etc.
  • Knowledge of Object-oriented programming
  • Experience with applications designed to display archived web content
  • Experience with AWS cloud services (EC2)
  • Python Tech stack (Python libraries - scrapy, requests, Urllib, Beautiful soup, splash, Selenium, pandas)

Responsibilities * Program and apply your knowledge set to fetch data from multiple online sources, cleanse it

  • Develop application frameworks for automating and maintaining constant flow of data from multiple sources
  • Design, build web crawlers to scrape data and URLs by using Python modules [scrapy, selenium, requests, Beautiful Soup, splash, etc.]
  • Create crawlers for all types of websites irrespective of the technical roadblocks.
  • Manage the crawlers to overcome technical challenges like IP ban, geolocation ban, captcha and bot blocking services
  • Design scrapy pipelines to connect the crawler output to MySQL database
  • Integrate the data crawled and scraped into our databases
  • Build and maintain high quality reusable code
  • Automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability

Job Type: Full-time

Salary: Up to ₹1,200,000.00 per year


  • Night shift