We are looking for a senior profile to extend and maintain the data collection pipeline that is at the heart of our business.
We are a data-driven company which collects and processes more than 500GB of raw data daily. We leverage big data technologies such as Serverless, Spark on AWS EMR to crunch these volumes of data and make it queryable.
In this role you will ensure that our data collection engine, which consists of distributed web crawlers is state of the art and ahead of our competition. You will ensure that we can scrape any webshop, no matter the ban-detection that has been put in place. Next to that it will be important that the proper monitoring tools are in place as we are scraping more than > 50 different webshops and that number is only going to increase.
As our senior software engineer you will be responsible for the following topics:
- Distributed web crawling architectures.
- Cost-effective data processing architectures.
- Advanced system monitoring solutions & dashboards.
- Design easy ways of interpreting scraped HTMLs.
- Advanced proxy management.
On top of all this you’ll make sure that Daltix stays competitive in terms of data collection by using the latest & most suitable technologies throughout our stack.
- You have a master in computer engineering (or relevant computer engineering degree).
- 5 years of relevant experience of which 3 years of programming experience in Python.
- Highly experienced in software engineering & design.
- Deep understanding of distributed systems.
- Able to lead software engineering projects as a tech lead.
- Highly proficient in spoken and written English.
- You are passionate about software engineering.
Ideally you also…
- Have experience building on top of Amazon Web Services.
- Have programming experience with Go.
- Experience with big data technologies (such as Hadoop, Spark, Airflow, Cassandra, Elasticsearch) is a plus.
- Have a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and security.
- Get energy from working in a highly complex and challenging startup environment with a high tech product.
WHAT CAN DALTIX OFFER YOU?
Daltix’ offers a competitive wage (including various benefits etc) and a young, dynamic and international (we have offices in Belgium and Portugal) atmosphere to work in.
You will also receive the possibility to work from home if you prefer (even if you live in Lisbon).
When you start working at Daltix, you will get a deep dive experience. You learn all you need to know about us, our journey, your future colleagues, the tools we work with, etc.
Going beyond, is coded in our company DNA. As soon as you start working, we expect a hands-on approach, with an entrepreneurial mentality.
You will also be able to participate in relevant trainings to stay at the top of this field.
Besides developing your technical skills you will also have the opportunity to grow into the following skill sets:
- Technical/architectural lead.
- SW project management.
- Team leading & coaching.
apply by sending your CV and cover letter (in English) to [email protected]