Add columns to Delimited File Connector in BuildMap Rule Error - Column Mismatch

[gmilunich (Geoff Milunich)]

Working with a Delimited File connector having some issues. I need to add a column to the data (why is irrelevant here) but assume it is a String type, either single or multi-valued.

Baseline:

• I have my CSV with fields: USERNAME, FULLNAME, STATUS, COL1, COL2, COL3
• I imported the schema from this file.
• I aggregated the file, which succeeded and I can see the records.

Adding the column and initial test.

• I added the schema attribute for the new column, COL2COL3, which is a string type.
• I added a BuildMap rule that concatenates COL2, “-”, COL3 so the final value is VAL2-VAL3 and puts it back into the map with the key COL2COL3, matching the schema attribute.
• I then do “Aggregate using Latest File” and it works, my new column COL2COL3 is populated correctly.

Aggregate New Files:

• I then make some updates to my file for testing and drag the new file into the box to aggregate the data.
• The system throws and error that it can not fine COL2COL3 in the data

It seems that this is the same error you get if you try to skip fields within a CSV file as well, except they are populated in the other list. It seems that this particular check is happening PRIOR to the BuildMap Rule running, which is causing the error. It seems that this should be happening AFTER the buildMap rule so that only the fields that need to be in ISC are loaded.

Retesting

• I then go back and do “Aggregate using Latest File” and it works, my new column COL2COL3 is populated correctly.

This tells me that the process works as expected, but the column name to schema check is holding back this functionality from working. This seems like a bug to me, rather than just how the product should work. JDBC BuildMap Rule I believe allows you to add columns. I have not checked IIQ’s Delimited File Connector yet

Some relevant posts related to this issue:

Connector: Integrating SailPoint with Delimited File Source

To ignore extra columns in th delimited file connector

there are extra columns in the delimited file that shouldn’t be imported. I am not able to find the way. Has anyone worked on a similar use case?

Delimited File BuildMap - Map > Making a new column possible?

Hi guys! Google is giving me a bunch of weird java examples that I’m having a hard time parsing. Requesting any idea’s here for being able to make a column at the end to show as an entitlement. We have a delimited file source with a couple columns built out via the Connector BuildMaps function. Is it possible to get this to make a new column at the end? Below is some snippets from the code that work correctly. Map map = DelimitedFileConnector.defaultBuildMap( cols, record ); String acces…

[rohit_wekhande]

@gmilunich

COL2COL3

Hello,

While aggregating the records from the CSV File, you are manually uploading the file into ISC? Correct? I think yes.

The Build Map Rule gets executed after initial checks are executed by ISC and are Success, after that, the Data gets into ISC and Build Map Rule Execution Starts.

That’s what I have observed when I was working with Flat File Connector or even with JDBC Connector in that care where we use Build Map Rules.

Regards,

Rohit Wekhande.

[gmilunich (Geoff Milunich)]

That is correct. I am manually adding the file to ISC through the UI (Drag and Drop) or through the API from Postman (to test unoptimized Aggregations.)

I agree that ISC Does it’s initial checks and then runs the BuildMap rule, but I am not sure why checking that the SCHEMA field match exactly the file fields is necessary as a pre-check. In the BuildMap Rule I can add or remove as many fields as I want and the system happily process it. The map of the data that comes into the BuildMap rule in fact only contains the fields that have data in them for that record.

Additionally, if I add a new Schema attribute after I have aggregated in a file, the system will happily re-aggregate the data without giving the error for mismatched columns.

It seems that this check is forcing users to either manually modify their data files before ingesting into ISC, or to include additional, unnecessary data in their systems.