Skip to content

Timeouts

Timeouts during execution

When executing long-running workflows or when loading or sending huge amounts of data, timeouts can occur, leading to execution errors. Timeouts can occur at different steps of the overall process:

  • If running the workflow via the test execution button, the frontend container nginx reverse proxy can time out. To adjust these timeouts the nginx.conf file must be modified. One way to do this is mounting a modified nginx.conf to the appropriate location into the service-container.
  • The backend / runtime uvicorn server can terminate executions: In a multi-worker process setup (i.e. WEB_CONCURRENCY>1) uvicorn kills and restarts worker processes not answering its health checks within the worker healthcheck timeout, which can happen during compute intensive executions. It is set via the UVICORN_TIMEOUT_WORKER_HEALTHCHECK environment variable on the backend / runtime service, defaulting to 30 (seconds). It is often necessary to increase this to allow handling compute intensive data science runtime executions.
  • Backend and runtime start uvicorn themselves, not via its command line interface. Hence the UVICORN_... environment variables of the uvicorn documentation have no effect, except for those hetida designer reads itself: UVICORN_TIMEOUT_WORKER_HEALTHCHECK, UVICORN_TIMEOUT_KEEP_ALIVE (see Keep-alive timeout) and UVICORN_LOG_LEVEL. Other uvicorn timeouts cannot be configured.
  • For the deprecated gunicorn mode (enabled by setting HETIDA_DESIGNER_PURE_UVICORN to false) the worker timeouts can be set via TIMEOUT and GRACEFUL_TIMEOUT environment variables on the backend / runtime service. See gunicorn documentation.
  • Requests to send or receive data from the runtime to the adapter or requests to execute workflows/components from the backend to the runtime can cause timeouts. These timeouts can be set via the EXTERNAL_REQUEST_TIMEOUT envirionment variables on the backend / runtime service.
  • If running in kubernetes using an ingress, the ingress may have its own timeout settings.

Keep-alive timeout

The backend / runtime keep idle HTTP connections open for 65 seconds (keep-alive timeout), so that clients can reuse them for further requests. Clients reusing connections must stop using an idle connection before this timeout. Otherwise the server may close the connection just as a request arrives on it, and the request fails without being processed, e.g. with an httpx.ReadError. The default idle expiry of clients like httpx (5s) ensures this. Generally clients interacting with runtime and backend should have their idle expiry strictly below the 65s.

The keep-alive timeout can be set via the UVICORN_TIMEOUT_KEEP_ALIVE environment variable, in the deprecated gunicorn mode via KEEP_ALIVE. If a reverse proxy or load balancer reusing connections sits in front of the backend / runtime, the keep-alive timeout should be strictly longer than its idle timeout.