SKOOR Documentation

Migration guide to Scripts in Skoor-8.1

This guide helps transfer existing converters to the new scripts introduced in SKOOR-8.1

Types of Scripts

The following types of Scripts are supported (Python, Talend, Custom/Shell).

General Info

These changes/notes are valid for all script types.

Environment variables can be saved in a .env file in root.
Uploading a source file isn’t necessary anymore to start a job.
If a file is passed to the script, it will be passed as an Environment variable SKOOR_SOURCE_FILE.

Talend scripts still expect the sourceFile as an attribute, so they haven’t been changed.

# Replace source_file = getArg("sourceFile") with the following 

#IN PYTHON
source_file = os.environ.get('SKOOR_SOURCE_FILE')

#IN SHELL 
source_file="$SKOOR_SOURCE_FILE"

Python scripts

For Python scripts, there is no need to wrap the python main.py with a shell script <converters_name>_run.sh anymore.
The entry point should be main.py preferably in root.
Required dependencies should be defined in a pyproject.toml

Poetry is used for dependency management.

To add new dependencies to the Project, include the required Package using Poetry’s supported dependency specification.

The name of an imported package in a Python script does not always match the package name required for download. For example, adding pyscopg2 to pyproject.toml does not work as the correctc package-name is psycopg2-binary. Use PyPI to find the correct package names.

The following example could be used for creating an initial pyproject.toml

[tool.poetry]
name = "PROJECT_NAME"
version = "0.1.0"
description = ""
authors = ["SKOOR GmbH <info@skoor.com>"]
package-mode = false

[tool.poetry.dependencies]
python = "^3.9"
et-xmlfile= "^2.0.0"
greenlet= "^3.1.1"
numpy= "^2.0.2"
openpyxl= "^3.1.5"
pandas= "^2.2.3"
psycopg2-binary= "^2.9.10"
python-dateutil= "^2.9.0.post0"
python-dotenv= "^1.0.1"
pytz= "^2024.2"
six= "^1.16.0"
sqlalchemy= "^2.0.36"
typing-extensions= "^4.12.2"
tzdata= "^2024.2"
requests= "^2.32.3"
cachetools= "^5.5.0"
jsonschema= "^4.23.0"
multicorn= "^0.1.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Talend converters

No migration changes are needed to Talend converters.