SailPoint File Upload Utility 4.0.0

Identity Security Cloud (ISC) ISC Discussion and Questions
, ,
1

Hello Developer Community,

I just wanted to let you know that after several weeks of testing, we’ve released SailPoint File Upload Utility 4.0.0!

Some key items:

New Aggregation APIs

As we’ve discussed in our previous post File Upload Utility Plans with API Deprecations, the biggest shift in this version is that we’ve moved away from older CC APIs and have moved towards newer versioned APIs. By using the new version of File Upload Utility, you should be able switch your processes over in advanced of any sort of CC API deprecations.

Source ID Formats

From discussions and even our previous FAQs, we’ve been discussing the differences in source ID formats since this is integral to how File Upload Utility knows which file goes to which source. This is best illustrated by example:

  • CC Source ID: 62867
  • V3 Source ID: 2c9180835d191a86015d27b0a2c31334

The good news is that if File Upload Utility detects an older CC Source ID, it does attempt to lookup the new V3 Source ID. This does result in additional overhead for this lookup, but it does provide a degree of backwards compatibility. Ideally, going forward we should be ideally using V3 Source IDs going forward.

For further details, see the section on File Naming Backward Compatibility with Older Source IDs

Additional Changes

In addition, we’ve done some clean-up and modernization of the libraries we’re using, as well as some general updates and better error handling. This does mean that the command line options have changed slightly.

For further details, see the Options section of the documentation.

In addition to these changes we’ve also moved to more modern Java JDK 11+. Theres been a lot questions around support, and I can confirm that we support and test with OpenJDK 11 and 17.

Ready to Download?

If you are ready to give it a go, head to the SailPoint File Upload Utility’s Releases page, and download it.

If you have any suggestions, issues, etc. feel free to discuss them on the Developer Community, and feel free to contribute to the future of File Upload Utility as well.

Thank your support!

Neil McGlennon
Principal Architect
SailPoint Technologies

13 Likes
2

@neil_mcglennon we have to migrate this from 3.0 but before doing this we have to moigrate our java also can you let us know what is the java version supported for 3.0 tool

3

Hi @schattopadhy - File Upload Utility requires Java Development Kit (JDK) 11 or higher to run. We build and test against OpenJDK 11 as well as OpenJDK 17.

For future reference the version is on the announcement and the documentation as well. :slight_smile:

Hope this helps!

4

I am trying out the new File Upload Utility 4.0.0

I am getting this error below, advise please.

SailPoint File Upload Utility

Version: 4.0.0
Date: 2024-05-01 9:00 CST
Docs: File Upload Utility

URL: https://usfca-sb.api.identitynow.com
Client ID: 0cfbb6b5-60e3-4a02-8076-6fce9f3f2974
Files: /home/htlam/idnw/batch1
ObjectType: account
Optimization: true
Recursive: true
Extensions: csv
Simulation: false
Verbose: true
Timeout: 10000

Checking credentials…
Analyzing directory: /home/htlam/idnw/batch1
Analyzing account file: 2c91808461336aa3016134cbb2d330c2-emp.csv
File [2c91808461336aa3016134cbb2d330c2-emp.csv]: detected with source ID reference [2c91808461336aa3016134cbb2d330c2].
File [2c91808461336aa3016134cbb2d330c2-emp.csv]: Submitting Account Aggregation: Source ID[2c91808461336aa3016134cbb2d330c2], Disable Optimization[false]
File [2c91808461336aa3016134cbb2d330c2-emp.csv]: Error: HTTP Code[500] Message Body[{“detailCode”:“500.0 Internal fault”,“trackingId”:“c30e7b5fd08b4feaac6b67deae5ee1e6”,“messages”:[{“locale”:“und”,“localeOrigin”:“REQUEST”,“text”:“An internal fault occurred.”},{“locale”:“en-US”,“localeOrigin”:“DEFAULT”,“text”:“An internal fault occurred.”}],“causes”:}]
Complete.

Elapsed time: 2 seconds
Files processed: 1

Success: 0

Error: 1
/home/htlam/idnw/batch1/2c91808461336aa3016134cbb2d330c2-emp.csv

Skipped: 0

Tue May 21 01:18:30 PM PDT 2024

5

Hi @huu11 - Sorry that you’re having issues. For reference, the File Upload Utility has documentation around how to handle various error types in the Troubleshooting section. For 500 Internal Server Error you can see here:

If you see a 500 Internal Server Error, this means there is a problem with the SailPoint Cloud processing itself. If you see this, it is best to reproduce the issue using the REST APIs and alert SailPoint Support.

This particular issues doesn’t seem to be a problem with File Upload Utility as it is functioning as intended - transport to the cloud. I would contact SailPoint Support to look into the issue.

6

Hi Neil, found the issue, but need your confirmation.

  1. If I use the API Management (Client ID and Secret Key) I get that Error: HTTP Code[500] . I want to use the API Managment way because if I get canned, then someone in my team can continue to run this file uploader in case my account is IAM is deleted.

  2. I then used the “Personal Access Tokens” way and it ran fine.

  3. So the only way to run this is by “Personal Access Tokens” and not the “API Management” way?

7

Hi @huu11 - Glad you found the issue, and I appreciate letting me know.

To answer your question - Yes, these APIs require Personal Access Tokens (PATs) in order to run. These APIs are audited so that we know who ran it, and require a certain authorization level in order to operate.

The main difference between the Client ID and Secret from API Management vs Client ID and Secrets from Personal Access Tokens is that the former doesn’t have a user nor the correct access rights associated with it. The later (PATs) have a user and access rights (scopes) associated with it. I believe we talk about this nuance in our authentication docs:

API endpoints that require user level permissions require the use of Personal access tokens (PATs). Correspondingly, the endpoints a personal access token (PAT) can call depends on the permissions of the user who generated it and the configuration of ISC.

Note: If an API Key is used then ISC API calls are made outside of the context of a user and some API calls will not work.

In this case, the API is failing because it doesn’t have a users’ context.

Back the problem that you are trying to solve, you can always create other identities in the system if so desired. You could create an identity called “File Upload Utility” and then give it a local login, access rights, and then generate the Personal Access Token. I personally would take this a step further and vault these credentials in a PAM or KMS system as well as regular credential rotations too.

Hope this helps!

8

Hi @neil_mcglennon,

When I try to import a CSV file using the FileUploadUtility from my PowerShell script, I receive a success message indicating that the file has been successfully imported to SailPoint.
However, when I check in SailPoint UI, aggregation not start on source, and I already given source id in file name.

could you please help us on this. why the accounts aggregation not start on source.

9

Hi @in_amolk - Unfortunately, I can’t help troubleshoot the issues you might be having with your aggregations themselves. That process happens inside Identity Security Cloud once the files are received by the API. The File Upload Utility merely discovers the files and sends them to the cloud via the supported APIs.

I might suggest running File Upload Utility with the -v option and see if you have any errors from the aggregation. The verbose messages should also indicate any error messages from the API too.

10

Hi @neil_mcglennon
I am trying to use version 4 and I am getting backward compatibility issues for source id. My source starts with 1839. I get below error when I run this utility. I raised support ticket 17 days ago and still waiting for their response. Any ideas why I am getting this error?

Analyzing account file: 1839xyz-hr-users.csv
File [1839xyz-hr-users.csv]: Detected with older source ID reference [1839xyz]. Attempting to resolve.

Building source file lookup table. Starting source iteration.
To avoid this scan, please switch to new Source IDs in your file names.
Older Source ID references will not be used in the future.

    File [1839xxx-hr-users.csv]: Unable to resolve old source ID reference [1839xxx] to new source ID reference. This file will be skipped.
    File [1839xxx-hr-users.csv]: Does not contain a valid source ID. Skipping...
11

Hi @peratkar - The behavior for lookups is explained here. You might consider running with --verbose to see the lookup table generated.

If you are interested in the lookup generation, you can see the source code here.

The lookups require 1) iterating through all sources, and 2) a source having the attribute cloudExternalId populated (from old CC source IDs). Some newer sources might not have this set.

So you can either:

  • Ensure your source has the old Source ID in cloudExternalId and rely on lookups
  • Or use the new Source ID in the file name.

The later is the most reliable way forward as there are no lookups, and newer sources won’t have an old Source ID going forward.

12

Thanks @neil_mcglennon. Issue was resolved when i added sp:scopes:all grant. New utility is able to resolve old source id and aggregation is successful

1 Like
13

Awesome! Glad to hear it is working for you!

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.