Unable to create flat file source via API

aceiss · March 15, 2023, 2:16pm

Thank again, Colin for your feedback. It really helps. We dont have out Partner IDN instance yet so trying to build stuff has been difficult. pushing to a clients instance but not being able to access it.

Anyway, further questions

I can create a source (non-auth) in IDN. Im creating it as Delimited File type. At current when im making a request to get the account schema, im getting this error

"text": "Only a source type, DelimitedFile, supports schema file download"

So im not sure if my creation json is set correctly. The error states type “DelimitedFile” and I have tried both “DelimitedFile” and “Delimited File”. When the create response comes back its the same as passed, but when I request the sources list, the new source always has a type of “Delimited File” with the space. Is this a problem? is it irrelevant?

this is the JSON I send to create a source

{
 "description": "Source for Aceiss application Aceiss DFile",
 "owner": {
 "type": "IDENTITY",
 "id": ".....",
 "name": "......"
 },
 "cluster": null,
 "accountCorrelationConfig": null,
 "accountCorrelationRule": null,
 "managerCorrelationMapping": null,
 "managerCorrelationRule": null,
 "beforeProvisioningRule": null,
 "schemas": [
 {
 "type": "CONNECTOR_SCHEMA",
 "id": "....",
 "name": "account"
 },
 {
 "type": "CONNECTOR_SCHEMA",
 "id": ".....",
 "name": "group"
 }
 ],
 "passwordPolicies": null,
 "features": [
 "DISCOVER_SCHEMA",
 "DIRECT_PERMISSIONS",
 "NO_RANDOM_ACCESS"
 ],
 "type": "Delimited File",
 "connector": "delimited-file-angularsc",
 "connectorClass": "sailpoint.connector.DelimitedFileConnector",
 "connectorAttributes": {
 "deleteThresholdPercentage": 10,
 "connectionType": "file",
 "healthy": true,
 "cloudDisplayName": "Aceiss DFile",
 "cloudExternalId": "211943",
 "connectorName": "Delimited File",
 "api": "aceiss",
 "beforeProvisioningRule": null,
 "since": "2023-03-13T15:04:03.484Z",
 "status": "SOURCE_STATE_UNCHECKED_SOURCE_NO_ACCOUNTS"
 },
 "deleteThreshold": 10,
 "authoritative": false,
 "healthy": true,
 "status": "SOURCE_STATE_UNCHECKED_SOURCE_NO_ACCOUNTS",
 "since": "2023-03-13T15:04:03.484Z",
 "connectorId": "delimited-file",
 "connectorName": "Delimited File",
 "connectionType": "file",
 "connectorImplementationId": "delimited-file",
 "managementWorkgroup": null,
 "id": ".....",
 "name": "Aceiss DFile",
 "created": "2023-03-13T15:04:03.484Z",
 "modified": "2023-03-13T15:04:05.227Z"
 },

Could this be a problem with the bulk csv upload as well?

It really should be trivial to create a source and upload accounts via apis. So im probably missing something simple. I hope.

colin_mckibben · March 21, 2023, 1:48pm

There’s a nuance to creating delimited file sources that you might have missed when reading the spec. It’s not very intuitive, but if you want to create a delimited file source you also have to supply the provisionAsCsv query param, which you can find in the query params section of the create source spec. In your example, when creating your flat file source, use the following URL:

POST https://{tenant}.api.identitynow.com/v3/sources?provisionAsCsv=true

aceiss · March 22, 2023, 4:42pm

@colin_mckibben
So that creates the Source correctly now and I’m able to retrieve the account schema.

This is what gets returned

idnullnamenullgivenNamenullfamilyNamenulle-mailnulllocationnullmanagernullgroups

Seems odd that the commas are replaced by null. Am I missing something else when creating the source?

I also get the same when using the API in the documentation

No big deal as the docs show it comma separated, so I’ll use that when building the CSV to send to loadAccounts. But I was curious.

aceiss · March 22, 2023, 4:56pm

Also, is there a reason that the bulk uploads, Schema, Accounts, Entitlements, only use files? Do you guys think you’ll ever expand to bulk upload via JSON? Maybe in the SDK?

colin_mckibben · March 22, 2023, 7:05pm

The API was designed for the UI, which is driven by uploading/downloading files. Bulk upload via API is something you can request as an idea in our ideas portal. The more votes it gets the more likely it will be implemented.

aceiss · March 23, 2023, 1:37pm

@colin_mckibben
So that creates the Source correctly now and I’m able to retrieve the account schema.

This is what gets returned

idnullnamenullgivenNamenullfamilyNamenulle-mailnulllocationnullmanagernullgroups

Seems odd that the commas are replaced by null. Am I missing something else when creating the source?

colin_mckibben · March 23, 2023, 2:37pm

how are you getting the account schema? What API?

aceiss · March 23, 2023, 4:03pm

GET https://{{tenant}}.api.identitynow.com/beta/sources/{{sourceId}}/schemas/accounts

colin_mckibben · March 23, 2023, 6:15pm

I think it’s because your source connectorAttributes object doesn’t specify the comma as the delimiter, so it defaults to null. When creating a flat-file source, make sure you specify the delimiter. Here’s an example of my source details. Look for the delimiter field under the connectorAttributes object. It looks like "delimiter": ",".

{
    "description": "Flat file for managing employees",
    "owner": {
        "type": "IDENTITY",
        "id": "2c91808475b4334b0175e1e005006401",
        "name": "SailPoint Services"
    },
    "cluster": null,
    "accountCorrelationConfig": {
        "type": "ACCOUNT_CORRELATION_CONFIG",
        "id": "2c918084838b1cc90183a9ced42916c3",
        "name": "Employees [source] Account Correlation"
    },
    "accountCorrelationRule": null,
    "managerCorrelationMapping": {
        "identityAttributeName": "identificationNumber",
        "accountAttributeName": "manager"
    },
    "managerCorrelationRule": null,
    "beforeProvisioningRule": null,
    "schemas": [
        {
            "type": "CONNECTOR_SCHEMA",
            "id": "2c9180887671ff8c01767b4671fc7d5f",
            "name": "account"
        },
        {
            "type": "CONNECTOR_SCHEMA",
            "id": "2c9180887671ff8c01767b4671fc7d60",
            "name": "group"
        }
    ],
    "passwordPolicies": null,
    "features": [
        "DISCOVER_SCHEMA",
        "NO_RANDOM_ACCESS",
        "DIRECT_PERMISSIONS"
    ],
    "type": "DelimitedFile",
    "connector": "delimited-file-angularsc",
    "connectorClass": "sailpoint.connector.DelimitedFileConnector",
    "connectorAttributes": {
        "mergeColumns": [
            "groups"
        ],
        "group.mergeRows": true,
        "group.delimiter": ",",
        "mergeRows": true,
        "group.filetransport": "local",
        "partitionMode": "disabled",
        "connectionType": "file",
        "group.host": "local",
        "group.indexColumn": "id",
        "file": "/tmp/source-account-2c9180887671ff8c01767b4671fb7d5e8915006813819050481.csv",
        "delimiter": ",",
        "deltaAggregation": null,
        "host": "local",
        "cloudExternalId": "17517",
        "indexColumns": [
            "id"
        ],
        "cloudIdentityProfileName": "Employees",
        "group.mergeColumns": [
            "entitlements",
            "groups",
            "permissions"
        ],
        "hasHeader": true,
        "filterEmptyRecords": true,
        "filetransport": "local",
        "deleteThresholdPercentage": 10,
        "group.filterEmptyRecords": true,
        "group.hasHeader": true,
        "group.partitionMode": "disabled",
        "cloudAuthoritativeSourcePrecedence": 20,
        "formPath": null,
        "group.columnNames": [
            "id",
            "name",
            "displayName",
            "created",
            "description",
            "modified",
            "entitlements",
            "groups",
            "permissions"
        ],
        "templateApplication": "DelimitedFile Template",
        "group.file": "/sweep-server/tmp/3297651b1fa245e294715d0e001ae664.csv",
        "indexColumn": "id",
        "healthy": true,
        "cloudDisplayName": "Employees",
        "connectorName": "Delimited File",
        "beforeProvisioningRule": null,
        "cloudOriginalApplicationType": "Delimited File",
        "since": "2022-11-22T15:12:17.208569Z",
        "status": "SOURCE_STATE_HEALTHY"
    },
    "deleteThreshold": 10,
    "authoritative": false,
    "healthy": true,
    "connectorId": "delimited-file",
    "connectorName": "Delimited File",
    "connectionType": "file",
    "connectorImplementationId": "delimited-file",
    "managementWorkgroup": null,
    "name": "Employees 2"
}

You can either create a new source with the delimiter set properly, or patch your existing source to add the delimiter attribute.

PATCH /v3/sources/{sourceId}

Body

[
    {
        "op": "add",
        "path": "/connectorAttributes/delimiter",
        "value": ","
    }
]

Topic		Replies	Views
Schema issues using Create Source API ISC Discussion and Questions	2	1478	July 19, 2023
Edit Configuration page on Delimited File Connector not working Bugs identity-security-cloud , flat-file-connector , connectors , fixed	6	308	May 19, 2024
Documentation on Creating source via API ISC Discussion and Questions	6	1798	July 19, 2023
Upload source entitlement schema API results in 400 ISC Discussion and Questions apis , flat-file-connector	16	800	July 11, 2023
Issue downloading IDN Delimited File Template from identitynow-demo tenants Bugs identity-security-cloud , flat-file-connector , sources	7	410	January 5, 2024

Unable to create flat file source via API

Related topics