Documentation

This guide helps transfer existing converters to the new converters introduced in SKOOR-9

Types of Converters

The following types of Converters are now supported (Python, Talend, Custom/Shell).

General Info

These changes/notes are valid for all converter types.

1. Environment variables can be saved in a .env file in root.

2. Uploading a source file isn’t necessary anymore to start a job.

3. If a file is passed to the script, it will be passed as an Environment variable SKOOR_SOURCE_FILE.

# Replace source_file = getArg("sourceFile") with the following 

#IN PYTHON
source_file = os.environ.get('SKOOR_SOURCE_FILE')

#IN SHELL 
source_file="$SKOOR_SOURCE_FILE"

Python converters

1. For Python converters, there is no need to wrap the python main.py with a shell script <converters_name>_run.sh anymore.

2. The entry point should be main.py preferably in root.

3. Required dependencies should be defined in a pyproject.toml

The following example could be used for creating the pyproject.toml

[tool.poetry]
name = "PROJECT_NAME"
version = "0.1.0"
description = ""
authors = ["SKOOR GmbH <info@skoor.com>"]
package-mode = false

[tool.poetry.dependencies]
python = "^3.9"
et-xmlfile= "^2.0.0"
greenlet= "^3.1.1"
numpy= "^2.0.2"
openpyxl= "^3.1.5"
pandas= "^2.2.3"
psycopg2-binary= "^2.9.10"
python-dateutil= "^2.9.0.post0"
python-dotenv= "^1.0.1"
pytz= "^2024.2"
six= "^1.16.0"
sqlalchemy= "^2.0.36"
typing-extensions= "^4.12.2"
tzdata= "^2024.2"
requests= "^2.32.3"
cachetools= "^5.5.0"
jsonschema= "^4.23.0"
multicorn= "^0.1.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Talend converters

No migration changes needed to Talend converters.