ETL system for maritime transporation

Country: Belgium

Our customer is an international organization that provides guidance in order to enhance the effectiveness and efficiency of Customs administrations. They developed a cargo manifest risk assessment and targeting solution to enable its Members to identify high risk cargo consignments and facilitate legitimate trade. Our mission was to develop an ETL component for the existing system that should enable administrators to map the system’s XML data fields to various data provider file formats. This implementation would allow the organization to focus completely on developing the actual targeting solution, instead of focusing on data management.

We developed a stand-alone Java Web application on the Spring Boot framework that allowed users to create dynamic mappings between input and output data. We stored the mappings in a MySQL database and a scheduled job would regularly check remote ftp locations for new files and map them based on the defined set of rules to the organization’s format. In the end, input files that were syntactically and semantically correct were transformed into the organization’s format and transferred to a remote SFTP server for further processing. Files that didn’t match the rules were rejected and a log was created in order to help administrators understand why the files were not correct. Supported data formats were XML and various UN/EDIFACT CUSCAR versions.

Team structure:

  • 1 software architect (15+ years experience)
  • 2 full-stack Java developers (7+ years experience)
  • 1 software tester (10+ years experience)

Time to completion: 9 months

Budget range: 75.000 € - 100.000 €

Technology stack:

  • Java 8
  • Spring Boot
  • Thymeleaf
  • XSLT
  • MySQL
  • Bash
  • VMWare vSphere

Comparison view

Development hours spent per month on data mapping

With ETL solution
20 hours -2000%

Customers onboarded per year

With the system
7 countries