ConnectorError with new connector that uses SaaS Connectivity framework

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Below is the account list command that works fine locally but fails on the tenant. I have played around with page size and it makes no difference. Connection error happens right after the first res.send( …) command. As you can see from error logs below code it only takes 2 sec to go thru the first loop so definitely not a timeout issue.
Connector aggregation stops with error right after ConnectionErorr occurs @ [2023-07-27T09:21:49.277-05:00], even thought as you can see in the logs aggregation continues.

 .stdAccountList(async (context: Context, input: StdAccountListInput, res: Response<StdAccountListOutput>)=> {
            const resultsCount = await myClient.getAllUsersCount()
            let offset = 0
            const pageSize = 10 
            logger.info(`stdAccountList resultsCount = ${resultsCount}`)
            while (offset < resultsCount) {
                logger.info(`offset ${offset}`)
                myClient.getAllUsersPaging(offset, pageSize).then((users) => {
                    users.forEach(user => {
                        logger.info(`processing ${user.firstName} account`)
                        res.keepAlive()
                        res.send({
                            identity: user.externalId ? user.externalId : '',
                            uuid: user.id ? user.id.toString() : '',
                            attributes: {
                                firstName: user.firstName ? user.firstName : '',
                                lastName: user.lastName ? user.lastName : '',
                                email: user.emails ? (user.emails.length > 0 ? user.emails[0].address : '') : '',
                            }
                        })
                    })
                }).catch(logger.error)
                offset += pageSize
            }
        })

  sail conn logs tail
        [2023-07-27T09:21:11.282-05:00] INFO  | updateTag        ▶︎ updated connector tag latest to version: 16
        [2023-07-27T09:21:47.030-05:00] INFO  | invokeCommand    ▶︎ Command execution started : std:account:list, for connector version 16.
        [2023-07-27T09:21:47.150-05:00] INFO  | invokeCommand    ▶︎ Command invocation complete : std:account:list, for connector version: 16. Elapsed time 120.336264ms
        [2023-07-27T09:21:47.737-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":16}
        [2023-07-27T09:21:47.847-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"stdAccountList resultsCount = 1023","version":16}
        [2023-07-27T09:21:47.847-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":16}
        [2023-07-27T09:21:47.850-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 0","version":16}
        [2023-07-27T09:21:49.263-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Joe account","version":16}
        [2023-07-27T09:21:49.264-05:00] INFO  | commandResponse  ▶︎ {"commandType":"std:account:list","completed":false,"elapsed":2104,"invocationId":"","message":"command response processed. output_count=0 keep_alive_count=1 state_count=0","requestId":"e0a2ed05-54c1-4244-8c05-84ae9819d265","version":null}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Andrew account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Jane account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Lidya account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Corina account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Alex account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Don account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Michelle account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Caroline account","version":16}
        [2023-07-27T09:21:49.267-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Prod Test account","version":16}
        [2023-07-27T09:21:49.277-05:00] INFO  | commandOutcome   ▶︎ {"commandType":"std:account:list","completed":true,"elapsed":2118,"error":"[ConnectorError] error receiving response from connector: stream client connection is broken (connector process may have crashed)","invocationId":"","message":"command failed","requestId":"e0a2ed05-54c1-4244-8c05-84ae9819d265","version":null}
        [2023-07-27T09:22:17.861-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 10","version":16}
        [2023-07-27T09:22:17.862-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":16}
        [2023-07-27T09:22:22.360-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Jim account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing David account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Prasad account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing William account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Josh account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Jacob account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Charlie account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Clare account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing Jack account","version":16}
        [2023-07-27T09:22:22.361-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"processing James account","version":16}
3

Have you checked if there is any CIDR whitelist defined in your end system that is not accepting a request from your tenant based on it’s IP address

4

I see what is happening here. Because you are running through the loop asynchronously, the while loop completes immediately, and the function is exited before any accounts are actually sent to IDN. Because of this, IDN thinks the connector has crashed. You need to loop through using await so that you don’t exit the method right away.

To do this, you can change the code to something like this:

 .stdAccountList(async (context: Context, input: StdAccountListInput, res: Response<StdAccountListOutput>)=> {
            const resultsCount = await myClient.getAllUsersCount()
            let offset = 0
            const pageSize = 10 
            logger.info(`stdAccountList resultsCount = ${resultsCount}`)
            while (offset < resultsCount) {
                logger.info(`offset ${offset}`)
                const users = await myClient.getAllUsersPaging(offset, pageSize);
                    users.forEach(user => {
                        logger.info(`processing ${user.firstName} account`)
                        res.send({
                            identity: user.externalId ? user.externalId : '',
                            uuid: user.id ? user.id.toString() : '',
                            attributes: {
                                firstName: user.firstName ? user.firstName : '',
                                lastName: user.lastName ? user.lastName : '',
                                email: user.emails ? (user.emails.length > 0 ? user.emails[0].address : '') : '',
                            }
                        })
                    })
                offset += pageSize
            }
        })
2 Likes
5

Thanks that definitely would be an issue, I have updated my code as suggested!
Problem is that error still exist, so I am suspecting more that one issue at play here.
I have to add that I had additional code where I am collecting large base 64 image string (~140 KB) right after email:
userImageBase64: user.userImageBase64 ? user.userImageBase64 : ‘’,
Only If I comment out above line everything now works fine and I am able to aggregate. I had played around and limited code to only aggregate 5 records (doing one per loop) and still same error happens right away. 5 records X 140 KB is well under limit of 200MB (connector memory) as stated in the documentation.

6

Are you able to attach the updated code with the image fetch? I am guessing that is somewhat related to the initial problem, but without the code it would be difficult to say what is wrong.

7

Here is the code with only 5 records that include base64 image string where error still happens

  .stdAccountList(async (context: Context, input: StdAccountListInput, res: Response<StdAccountListOutput>) => {
            logger.debug('listing accounts')
            const resultsCount = 5 //await myClient.getAllUsersCount()
            let offset = 0
            const pageSize = 1
            logger.info(`stdAccountList resultsCount = ${resultsCount}`)
            while (offset < resultsCount) {
                logger.info(`offset ${offset}`)
                const users = await myClient.getAllUsersPaging(offset, pageSize);
                users.forEach(user => {
                    //logger.info(`processing ${user.firstName} account`)
                    res.send({
                        identity: user.id ? user.id.toString() : '', 
                        uuid: user.id ? user.id.toString() : '',
                        disabled: user.suspended,
                        attributes: {
                            id: user.id,
                            externalId: user.externalId ? user.externalId : '',
                            firstName: user.firstName ? user.firstName : '',
                            lastName: user.lastName ? user.lastName : '',
                            email: user.emails ? (user.emails.length > 0 ? user.emails[0].address : '') : '',
                            userImageBase64: user.userImageBase64 ? user.userImageBase64 : ''
                        }
                    })
                })
                offset += pageSize
            }
        })


[2023-07-31T09:20:30.531-05:00] INFO  | invokeCommand    ▶︎ Command execution started : std:account:list, for connector version 64.
[2023-07-31T09:20:30.589-05:00] INFO  | invokeCommand    ▶︎ Command invocation complete : std:account:list, for connector version: 64. Elapsed time 57.711566ms
[2023-07-31T09:20:31.201-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"stdAccountList resultsCount = 5","version":64}
[2023-07-31T09:20:31.203-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":64}
[2023-07-31T09:20:31.242-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 0","version":64}
[2023-07-31T09:20:31.444-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 1","version":64}
[2023-07-31T09:20:31.444-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":64}
[2023-07-31T09:20:31.739-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 2","version":64}
[2023-07-31T09:20:31.740-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":64}
[2023-07-31T09:20:32.081-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 3","version":64}
[2023-07-31T09:20:32.081-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":64}
[2023-07-31T09:20:32.288-05:00] INFO  | commandOutcome   ▶︎ {"commandType":"std:account:list","completed":true,"elapsed":1690,"error":"[ConnectorError] error receiving response from connector: stream client connection is broken (connector process may have crashed)","invocationId":"","message":"command failed","requestId":"ca698351-1254-46ae-97bd-e1a2396e5ce0","version":null}
[2023-07-31T09:20:32.288-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"offset 4","version":64}
[2023-07-31T09:20:32.288-05:00] INFO  | connectorMessage ▶︎ {"commandType":"std:account:list","invocationId":"","message":"-------------------------------------------AxiosWrapper created\n","version":64}

If I comment out image related line it works fine…

8

I wonder if it has to do with the foreach loop, which is also asynchronous in javascript.

Try changing the loop to a for of loop like the following:

for (const user of users) {
    res.send({<object>})
}
9

here is the latest code as suggested …same error:

       .stdAccountList(async (context: Context, input: StdAccountListInput, res: Response<StdAccountListOutput>) => {
            logger.debug('listing accounts')
            const resultsCount = 5 //await myClient.getAllUsersCount()
            let offset = 0
            const pageSize = 1
            logger.info(`stdAccountList resultsCount = ${resultsCount}`)
            while (offset < resultsCount) {
                logger.info(`offset ${offset}`)
                const users = await myClient.getAllUsersPaging(offset, pageSize);
                for  (const user of users) { //users.forEach(user => {
                    //logger.info(`processing ${user.firstName} account`)
                    res.send({
                        identity: user.id ? user.id.toString() : '', //user.emails ? (user.emails.length > 0 ? user.emails[0].address :  (user.id ? user.id.toString() : '')) : (user.id ? user.id.toString() : ''),
                        uuid: user.id ? user.id.toString() : '',
                        disabled: user.suspended,
                        attributes: {
                            id: user.id,
                            externalId: user.externalId ? user.externalId : '',
                            firstName: user.firstName ? user.firstName : '',
                            lastName: user.lastName ? user.lastName : '',
                            email: user.emails ? (user.emails.length > 0 ? user.emails[0].address : '') : '',
                            userImageBase64: user.userImageBase64 ? user.userImageBase64 : ''
                        }
                    })
                }
                //)
                offset += pageSize
            }
        })
10

additional details - when I am testing locally account read I am getting error below. I wonder if this is related to connector error that I get on the tenant. I get correct output with error at the end. If I comment out userImageBase64 line everything works fine.

...
      .stdAccountRead(async (context: Context, input: StdAccountReadInput, res: Response<StdAccountReadOutput>) => {
            const user = await myClient.getUser(input.identity)
            console.log(`Processing  ${JSON.stringify(user)}`)
            res.send({
                identity: user.id ? user.id.toString() : '', //user.emails ? (user.emails.length > 0 ? user.emails[0].address :  (user.id ? user.id.toString() : '')) : (user.id ? user.id.toString() : ''),
                uuid: user.id ? user.id.toString() : '',
                disabled: user.suspended,
                attributes: {
                    id: user.id,
                    externalId: user.externalId ? user.externalId : '',
                    firstName: user.firstName ? user.firstName : '',
                    lastName: user.lastName ? user.lastName : '',
                    email: user.emails ? (user.emails.length > 0 ? user.emails[0].address : '') : '',
                    userImage: user.userImage ? user.userImage.uri : '',
                    userImageBase64: user.userImageBase64 ? user.userImageBase64 : '',
                },
            })
            logger.info(`stdAccountRead read account : ${input.identity}`)

        })
....
....
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
    at new NodeError (node:internal/errors:405:5)
    at ServerResponse.<anonymous> (node:internal/streams/pipeline:411:14)
    at ServerResponse.emit (node:events:524:35)
    at emitCloseNT (node:_http_server:1012:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21) {
  code: 'ERR_STREAM_PREMATURE_CLOSE'
}

Based on debugging above error locally it happens in spcx.js file on connector-sdk ( “@sailpoint/connector-sdk”: “^1.1.3”), specifically on res.end() call

const app = express_1.default()
        .use(express_1.default.json({ strict: true }))
        .post('/*', async (req, res) => {
        try {
            const cmd = req.body;
            await lib_1._withConfig(cmd.config, async () => {
                const connector = await loadConnector(connectorPath);
                const out = new stream_1.Transform({
                    writableObjectMode: true,
                    transform(chunk, encoding, callback) {
                        try {
                            this.push(JSON.stringify(chunk) + '\n');
                        }
                        catch (e) {
                            callback(e);
                        }
                        callback();
                    },
                });
                stream_1.pipeline(out, res.status(200).type('application/x-ndjson'), (err) => {
                    if (err) {
                        console.error(err);
                    }
                });
                await connector._exec(cmd.type, { version: cmd.version, commandType: cmd.type }, cmd.input, out);
                out.end();
            });
        }
        catch (e) {
            console.error(e?.message || e);
            let errorType = lib_1.ConnectorErrorType.Generic;
            if (e instanceof lib_1.ConnectorError) {
                errorType = e.type;
            }
            res.status(500).send(`${errorType} error: \n + ${util_1.inspect(e)}`);
        }
        finally {
            res.end();
        }
    });
    app.listen(port, () => {
        console.log(`SailPoint connector local development server listening at http://localhost:${port}`);
    });
11

You should get a response in postman when testing locally that has more details about the error than what shows up in the console. Can you look at that data? It likely has to do with the type of the base64 value.

You can also debug the code and step through manually using the steps listed here: Debugging | SailPoint Developer Community which might point you to the right direction.

Thanks,
Phil

12

Unfortunately I no longer see any ERR_STREAM_PREMATURE_CLOSE (or any other error) locally and I have not done any coding changes. After redeploying my local code to the tenant I still see error on the tenant as before:

{"detailCode":"Internal Server Error","trackingId":"f86225a3-7f79-4140-8a7c-ea1a7df44709","messages":[{"locale":"en-US","localeOrigin":"DEFAULT","text":"[ConnectorError] error receiving response from connector: stream client connection is broken (connector process may have crashed)"}]}

and as before if I comment out line that send base64 image it works fine;
//userImageBase64: user.userImageBase64 ? user.userImageBase64 : '',

13

Each response object is limited to 256kb due to the queue that is used to store the response. I think this could be the reason why the connector is failing when you try to send back an image. Can you check to see the size of the entire object? I think the better way to handle will be to store that image in some other place and reference the URL instead of trying to embed directly in IdentityNow.

14

I was actually able to reproduce this issue on my end. I think this long message is a bug in the SDK where it isn’t awaiting for the end of each res.send to complete before exiting the function. There should be a fix for this soon, but in the meantime, a simple workaround is to simply add a line at the end of the command like this:

return new Promise(resolve => setTimeout(resolve, 1000))
15

Thanks for both suggestions:

  1. “Bug in SDK around res.send” adding 1 sec wait resolved issue when I test locally :slight_smile:
  2. “256kb limit in tenant queue” - our data size is ~1MB per record so this explains why even single account aggregation fails on the tenant. Suggest to specify that on online doc.
    We will go back and try to build out of band process to get image from vendor and store it in some S3 bucket. Primary issue is that this will only fix half of the problem as we still need to upload image to vendor site during onboarding event. I am suspecting we will run into same issue on create account event (or again have to build out of band process)
16

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.