Delimited File source - File Settings - Bug?

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Hi,

I dont know if this is a bug or how the product is meant to work,

I created a Delimited File Source.

Under File Settings > Account > Filtering, set ‘Number of Lines to Skip’ to 4

as my input file looks like:

Acme Inc
User Report
------
Name,Organisation,Email
Joe Biggs,Acme,joe@acme.com
Jane Doe,Acme,jane@acme.com

The file import (using the UI or /source/:source/load-accounts API) fails, as it does not skip the 4 lines before trying to import the accounts.

It is trying to read the columns from Row 1, before skipping.

The error returned is

“Column:"[Acme Inc]" is unknown and/or column: "[Name, Organisation, Email]" is missing”

Is this correct behaviour? Is there an attribute I can set on the source JSON to tell it to skip before trying to parse the data?

If I manually remove the 4 rows from the input file, then aggregation is successful except ISC skips the first 4 accounts :stuck_out_tongue:

Thank you

2

Hello Jason,

I dont think this is a bug, this looks like the expected behavior. The ISC Delimited File connector requires the first line of the file to be column headers, and Number of Lines to Skip skips data rows after the header rather than lines from the top of the file (Delimited File Prerequisites).

image
image1972×626 62 KB

You can move Name,Organisation,Email to line 1, keep the other rows (Acme Inc, User Report, ------) right below it, and set Number of Lines to Skip = 3. ISC reads the header, skips the next 3 rows, and aggregates from Joe Biggs onward.

Or
move the header(Name,Organisation,Email) to line 1 and use a # in front of the other 3 lines. The connector ignores anything starting with # (it’s the default comment character), so those rows get skipped automatically.

3

Ahh, thank you for taking the effort to clarify this.