Salary: Negotiable
Age: 21-35 years old
Working hours: 9 hours a day, 4 days off per month
Job Requirements:
1. Bachelor's degree or above, majoring in Computer Science or related fields, with 3+ years of experience in Python web scraping preferred.
2. Familiar with the Scrapy framework and its principles, with experience in large-scale distributed development and deployment.
3. Proficient in web/app common anti-scraping mechanisms, and skilled in various anti-scraping techniques including, but not limited to: hook, JS protection, obfuscation, reverse analysis, app anti-cracking, reverse engineering protection, hardening techniques, etc.
4. Proficient in HTTP protocol, XPath, CSS path, and regular expressions.
5. Familiar with Selenium or Puppeteer.
6. Familiar with common anti-scraping methods and corresponding techniques.
7. Strong sense of responsibility, good communication skills, learning ability, and teamwork spirit.
8. Experience in analyzing and debugging common JS protection techniques is a plus.
9. Experience with overseas website scraping, penetration, API interfaces, database migration, and unstructured data is a plus.
Responsibilities:
1. Design, develop, maintain, and refactor standalone and distributed web crawlers.
2. Responsible for data scraping, cleaning, structuring, and storing, and continuous operation and maintenance.
3. Update and maintain scraping strategies and ensure data extraction accuracy and efficiency.
4. Able to independently deploy crawling programs and maintain crawling servers.
5. Optimize crawler performance and stability.
6. Write development documentation to support subsequent development by other developers.