SaaS custom connector, timeout issues

[derekbotbyl]

I am currently running into two issues with a custom connector, both around timeouts.

Problem 1.

Scenario: Test Connection
Simply put, test no longer works. I can see in the logs that everything is successful and finishes with plenty of time, however ISC will wait until the end of the timeout and error.

Command completed: std:test-connection, for connector version 51.output_count=1 keep_alive_count=0 state_count=0. Elapsed time 12480ms

This process took under 30 seconds to run, yet hit a timeout.

I am sending one response per ‘connection’ I have established in the connector.

		.stdTestConnection(
			async (
				context: Context,
				input: StdTestConnectionInput,
				res: Response<StdTestConnectionOutput>,
			) => {
				// Log that we're going a thing
				logger.info("Running test connection");

				// Check each client configured
				for (const connection of config.connections) {
					logger.info(`Testing connection ${connection.connectionName}`);
					// Our custom HTTP client for Datalake
					const datalakeClient: DatalakeClient = new DatalakeClient(connection);

					// Get our token for auth
					await datalakeClient.establishConnection();

					// Run test connection
					res.send(await datalakeClient.testConnection());
				}
			},
		)

Problem 2

Scenario: List accounts
The endpoint I am utilizing can take a long time to respond. If I don’t send keepAlive this will cause a timeout. That is understood and I can work my way around it. What I was noticing however, is the process would retry, but the original process never terminated. This ends up with the process running up to 3 times, and none of them report back to ISC.
Is this something I need to account for, or is ISC simply not built to try and terminate any threads from timeout processes?

[sushantkulkarni]

@derekbotbyl For #1, can you ensure that your stdTestConnection function is properly returning a promise that resolves when all the test connections have been checked? This should inform ISC that the function has completed its task.

For #2, looks like you might have to account for it in code. Consider implementing a cancellation token (like in C#) or similar mechanism in your code. When a new process starts, you could use this token to cancel any previous processes that are still running. This would prevent multiple concurrent processes from running and ensure that only the latest process is active.

[derekbotbyl]

Thank you for your assistance Sushant. I’m fairly new to the concept of promises, however I do believe I have it set up to return a promise.

from my http client

	async get(url: string): Promise<AxiosResponse> {
		logger.info(`Getting ${this.client.defaults.baseURL}/${url}`)
		return this.client.get(url).then((response) => response);
	}

to my datalake client

	async testConnection(): Promise<object> {
		await this.httpClient.get("purged?limit=1").then((response) => {
			return {};
		});
		return {};
	}

to the connector

		.stdTestConnection(
			async (
				context: Context,
				input: StdTestConnectionInput,
				res: Response<StdTestConnectionOutput>,
			) => {
				// Log that we're doing a thing
				logger.info("Running test connection");

				// Check each client configured
				for await (const connection of config.connections) {
					logger.info(`Testing connection ${connection.connectionName}`);
					// Our custom HTTP client for Datalake
					const datalakeClient: DatalakeClient = new DatalakeClient(connection);

					// Get our token for auth
					await datalakeClient.establishConnection();

					// Run test connection
					res.send(await datalakeClient.testConnection());
				}
			},
		)

I did add an ‘await’ to the for loop in my stdTestConnection, but that did not have any impact. Can you help identify what I might be missing in these three code snips?

I’ll do some research on cancellation tokens. Essentially do I just set a variable on the connector, with a unique token for each run, and have the process check “is this my token, yes continue, no halt”?

[derekbotbyl]

I removed the res.send for each of the testConnection method calls, and added the following lines at the end.

				const successful = new Promise(() =>{
					logger.info("Completed connection tests")
				})
				res.send(successful)

This way it only sends a single promise after the tests are successful.
My connector tests are now passing again. I think the issue was that previously I was returning a single res.send, and then I changed my code to look at multiple connections which returned multiple res.send.