Your first integration
The server and workers are up and running, but are useless without integrations.
Registering the integration
Integrations are registered in the file dedicated located at
BLACKSMITH_DIR/integrations.BLACKSMITH_EXT. For the purpose of these guides,
we'll register a SQL integration using the PostgreSQL driver. It is the container
in the Docker stack previously started.
Every integrations are registered under the
integrations key, mapped by their
sql allows to register multiple SQL integrations. Everyone
one of them must have a unique name.
The minimal configuration for a SQL integration is as follow:
integrations: sql: - name: "warehouse" driver: "postgresql" connection: "ANY_ENV_VAR"
The connection string to connect to the SQL store is retrieved from an environment
variable. Here, the connection string of the integration
warehouse is the value
of the environment variable
In addition to a required
name and integration-specific configuration, an
integration can have
integrations: sql: - name: "warehouse" driver: "postgresql" connection: "ANY_ENV_VAR" policies: load: timeout: 20 initialInterval: 30 backoffCoefficient: 2 maximumInterval: 3600 maximumAttempts: 50
load policies are applied when Loading data into the integration. They
are as follow:
timeout(in seconds) is the maximum time of a single data Load execution attempt. This timeout should be as short as the longest possible execution of the Load.
initialInterval(in seconds) is the backoff interval for the first retry.
backoffCoefficientis the coefficient used to calculate the next retry backoff interval. The next retry interval is previous interval multiplied by this coefficient. Must be
maximumInterval(in seconds) is the maximum backoff interval between retries. Exponential backoff leads to interval increase. This value is the cap of the interval. If not interval is provided, the value is
100 * initialInterval.
maximumAttemptsis the maximum number of attempts of retries. When exceeded the retries stop even if not expired yet.
Re-starting the workers
Now that the integration is registered, we can re-start the workers so they are aware of the changes we've just made.
Note that we do not need to restart the server! This way, the server is always up to avoid connections loses with the clients as much as possible.
We now have a ready-to-use integration. How can we Load data into it?
If you notice something we've missed or could be improved on, please follow this link and submit a pull request to the repository. Once we merge it, the changes will be reflected on the website the next time it is deployed.
Thank you for your contributions!