File Upload Utility

[neil_mcglennon]

	Description	The SailPoint File Upload Utility provides a way for SailPoint cloud customers to upload files to the cloud automatically, for automatic account and entitlement aggregations.
	Legal Agreement	By using this CoLab item, you are agreeing to SailPoint’s Terms of Service for our developer community and open-source CoLab.
	Repository Link	GitHub - sailpoint-oss/colab-file-upload-utility
	Supported by	Community Developed
	Current Release	File Upload Utility 4.1.0

Overview

The SailPoint File Upload Utility provides a way for SailPoint customers to upload files to Identity Security Cloud automatically, for automatic account and entitlement aggregations. This is an executable Java-based JAR which can be executed with a set of options, or scheduled via CLI utilities like cron . This utility uploads files the same way as if you were to upload the files via the admin user interface, by calling the supported SailPoint REST APIs.

Getting Started

Downloading

The SailPoint File Upload Utility 4.1.0 is available via the Releases page.

Requirements

• Java - is a Java based application and requires Java Development Kit (JDK) 11 or higher to run. We build and test against OpenJDK 11 as well as OpenJDK 17.
• Network - This also requires external, outbound access over HTTPS (443) via REST API calls to the SailPoint Cloud as indicated by the URL you configure the File Upload Utility.
• ISC Personal Access Token (PAT)
  • User must have at least the Source Admin user level
  • The PAT must have at least the idn:sources:manage and idn:entitlements:manage scopes.

Usage

Using File Upload Utility is pretty straight-forward, and follows the pattern of normal Java JAR execution:

java -jar sailpoint-file-upload-utility.jar <options>

Options

We cover using specific use cases further below, but here are all available options to pass to the File Upload Utility.

Option	Required	Example Usage	Description
-u <arg>, --url <arg>	Required	--url https://example.api.identitynow.com	SailPoint API Gateway (e.g. https://tenant.api.identitynow.com)
-i <arg>, --clientId <arg>	Required	--clientId d0b...574	SailPoint Client ID (PAT). If value of env is provided, then the value for environment variable SAIL_CLIENT_ID will be used.
-s <arg>, --clientSecret <arg>	Required	--clientSecret a34...1df	SailPoint Client Secret (PAT). If a value is not supplied, will be prompted interactively. If value of env is provided, then the value for environment variable SAIL_CLIENT_SECRET will be used.
-f <arg>, --file <arg>	Required	--file /Users/neil.mcglennon/test/resources/	File or directories for bulk aggregation. This can be specified multiple times.
-d, --disableOptimization	Optional	--disableOptimization	Disable Optimization on Account Aggregation
-o <arg>, --objectType <arg>	Optional	--objectType group	File Type; Account or Entitlement Schema. Default: Account
-R, --recursive	Optional	--recursive	Recursively search directories
-S, --simulate	Optional	--simulate	Simulation Mode. Scans for files but does not aggregate.
-t <arg>, --timeout <arg>	Optional	--timeout 100000	Timeout (in milliseconds). Default: 10000 (10s)
-x <arg>, --extension <arg>	Optional	--extension csv	File extensions to search (for directories only). Default: csv
-v, --verbose	Optional	--verbose	Verbose logging. Default: false
-H <arg>, --proxyHost <arg>	Optional	--proxyHost proxy.host.com	Proxy host name or IP. Use --proxyHost and --proxyPort together.
-P <arg>, --proxyPort <arg>	Optional	--proxyPort 443	Proxy port. Use --proxyHost and --proxyPort together.
-U <arg>, --proxyUser <arg>	Optional	--proxyUser foo	Proxy user for authenticated proxies. Use --proxyUser and --proxyPassword together.
-W <arg>, --proxyPassword <arg>	Optional	--proxyPassword bar	Proxy password for authenticated proxies. Use --proxyUser and --proxyPassword together. If a value is not supplied, will be prompted interactively.
-V, --version	Optional	--version	Displays the current version.
-h, --help	Optional	--help	Displays help.

Guide

Help and Usage

In order to see help and usage of the File Upload Utility, supply the --help or -h options. The output of this looks like this:

$ java -jar sailpoint-file-upload-utility.jar --help
Usage:

Perform bulk file aggregations to Identity Security Cloud.

java -jar sailpoint-file-upload-utility.jar [-dhRSvV] [-s[=<clientSecret>]] [-W
[=<proxyPassword>]] [-H=<proxyHost>] -i=<clientId> [-o=<objectType>]
[-P=<proxyPort>] [-t=<timeout>] -u=<url> [-U=<proxyUser>] -f=<files>
[-f=<files>]... [-x=<fileExtensions>]...

Description:

Scans specified files and directories for files in bulk, to send to Identity
Security Cloud for account or entitlement aggregation.  For more details see:
https://developer.sailpoint.com/discuss/t/file-upload-utility/18181

Options:
  -u, --url=<url>           SailPoint API Gateway (e.g. https://tenant.api.
                              identitynow.com)
  -i, --clientId=<clientId> SailPoint Client ID (PAT)
  -s, --clientSecret[=<clientSecret>]
                            SailPoint Client Secret (PAT)
  -f, --file=<files>        File or directories for bulk aggregation.
  -d, --disableOptimization Disable Optimization on Account Aggregation
  -o, --objectType=<objectType>
                            File Type; Account or Entitlement Schema. Default:
                              Account
  -R, --recursive           Recursively search directories
  -S, --simulate            Simulation Mode.  Scans for files but does not
                              aggregate.
  -t, --timeout=<timeout>   Timeout (in milliseconds). Default: 10000 (10s)
  -x, --extension=<fileExtensions>
                            File extensions to search (for directories only).
                              Default: csv
  -v, --verbose             Verbose logging. Default: false
  -H, --proxyHost=<proxyHost>
                            Proxy Host
  -P, --proxyPort=<proxyPort>
                            Proxy Post
  -U, --proxyUser=<proxyUser>
                            Proxy User; Used for authenticated proxies
  -W, --proxyPassword[=<proxyPassword>]
                            Proxy Password; Used for authenticated proxies
  -V, --version             Displays the current version.
  -h, --help                Display help.

This command is useful for quick reference in places where documentation is not readily available.

Version

To see the current version of File Upload Utility, supply the --version or -V option. Output should look like this:

$ java -jar sailpoint-file-upload-utility.jar --version
SailPoint File Upload Utility 4.1.0
Build: 2024-09-10 16:15 CST
Documentation: https://developer.sailpoint.com/discuss/t/file-upload-utility/18181
JVM: 17.0.10 (Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.10+7-LTS)
OS: Mac OS X 14.6.1 aarch64

This command is useful for quick reference which version of the File Upload Utility you are running, as well as which version of Java is being run, often for troubleshooting processes.

Aggregating Account Files

To aggregate account files, you’ll want to specify the following options:

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/resources/ -R
------------------------------------------------------------------------------------------------------------
 SailPoint File Upload Utility
------------------------------------------------------------------------------------------------------------
 Version:            4.1.0                          
 Date:               2024-09-10 16:15 CST           
 Docs:               https://developer.sailpoint.com/discuss/t/file-upload-utility/18181 
------------------------------------------------------------------------------------------------------------
 URL:                https://example.api.identitynow.com 
 Client ID:          d0b28ce1b2694b64949dd546dd1ff574 
 Files:              /Users/neil.mcglennon/test/resources 
 ObjectType:         account                        
 Optimization:       true                           
 Recursive:          true                           
 Extensions:         csv                            
 Simulation:         false                          
 Verbose:            false                          
 Timeout:            10000                          
------------------------------------------------------------------------------------------------------------
Checking credentials...
Analyzing directory: /Users/neil.mcglennon/test/resources
Analyzing account file: 184744-AuthEmployees.csv
	File [184744-AuthEmployees.csv]: Aggregated successfully.
Analyzing account file: 2c918087701c40cf01701dfdf2c61e2a-AuthEmployees.csv
	File [2c918087701c40cf01701dfdf2c61e2a-AuthEmployees.csv]: Aggregated successfully.
Analyzing account file: 184744 - AuthEmployees.csv
	File [184744 - AuthEmployees.csv]: Aggregated successfully.
Analyzing account file: 81260-entitlement.csv
	File [81260-entitlement.csv]: Unable to resolve old source ID reference [81260] to new source ID reference. This file will be skipped.
	File [81260-entitlement.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file: 82343-entitlements.csv
	File [82343-entitlements.csv]: Unable to resolve old source ID reference [82343] to new source ID reference. This file will be skipped.
	File [82343-entitlements.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file: Don't Read.csv
	File [Don't Read.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file: f1f3b747be924745afbc0c8a53f71baf-file-test-account-feed.csv
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-account-feed.csv]: Aggregated successfully.
Analyzing account file: 82343-acounts.csv
	File [82343-acounts.csv]: Unable to resolve old source ID reference [82343] to new source ID reference. This file will be skipped.
	File [82343-acounts.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file: 82343-roles.csv
	File [82343-roles.csv]: Unable to resolve old source ID reference [82343] to new source ID reference. This file will be skipped.
	File [82343-roles.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file: f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: Error: HTTP Code[400] Message[] Body[{"detailCode":"400.1.3 Illegal value","trackingId":"edf48ac42d4a426384dcf0d0196cbfeb","messages":[{"locale":"en-US","localeOrigin":"DEFAULT","text":"Column: \"[displayName, created, description, modified, entitlements, permissions]\" is unknown and/or column: \"[givenName, familyName, e-mail, location, manager]\" is missing"},{"locale":"und","localeOrigin":"REQUEST","text":"Column: \"[displayName, created, description, modified, entitlements, permissions]\" is unknown and/or column: \"[givenName, familyName, e-mail, location, manager]\" is missing"}],"causes":[]}]
Analyzing account file: 81260-entitlement.csv
	File [81260-entitlement.csv]: Unable to resolve old source ID reference [81260] to new source ID reference. This file will be skipped.
	File [81260-entitlement.csv]: Does not contain a valid source ID. Skipping...
Analyzing account file:  184745 - AuthContractors.csv
	File [ 184745 - AuthContractors.csv]: Unable to resolve old source ID reference [184745] to new source ID reference. This file will be skipped.
	File [ 184745 - AuthContractors.csv]: Does not contain a valid source ID. Skipping...
Complete.
------------------------------------------------------------------------------------------------------------
 Elapsed time:       7 seconds                      
 Files processed:    12                             
------------------------------------------------------------------------------------------------------------
 Success:            4                              
------------------------------------------------------------------------------------------------------------
 Error:              1                              
------------------------------------------------------------------------------------------------------------
 Skipped:            7                              
------------------------------------------------------------------------------------------------------------

Aggregating Entitlement Files

To aggregate entitlement files, you’ll want to specify the following options:

Execution of this might look like the following:

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/resources/entitlements/f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv --objectType group
------------------------------------------------------------------------------------------------------------
 SailPoint File Upload Utility
------------------------------------------------------------------------------------------------------------
 Version:            4.1.0                          
 Date:               2024-09-10 16:15 CST           
 Docs:               https://developer.sailpoint.com/discuss/t/file-upload-utility/18181 
------------------------------------------------------------------------------------------------------------
 URL:                https://example.api.identitynow.com 
 Client ID:          d0b28ce1b2694b64949dd546dd1ff574 
 Files:              /Users/neil.mcglennon/test/resources/entitlements/f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv 
 ObjectType:         group                          
 Recursive:          true                           
 Extensions:         csv                            
 Simulation:         false                          
 Verbose:            true                           
 Timeout:            10000                          
------------------------------------------------------------------------------------------------------------
Checking credentials...
Analyzing group file: /Users/neil.mcglennon/test/resources/entitlements/f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv
Analyzing group file: f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: detected with source ID reference [f1f3b747be924745afbc0c8a53f71baf].
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: Submitting Entitlement Aggregation: Source ID[f1f3b747be924745afbc0c8a53f71baf], Object Type[group]
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: Aggregation response: {"type":"QUARTZ","id":"8f7ec314b9dc47abbbea347398826e07","name":"Cloud Group Aggregation","description":"Aggregates from the specified application.","parentName":null,"launcher":"slpt.services","created":"2024-09-11T02:11:51.912+00:00","launched":null,"completed":null,"progress":null,"completionStatus":null,"messages":[],"returns":[{"displayLabel":"TASK_OUT_ACCOUNT_GROUP_AGGREGATION_APPLICATIONS","attributeName":"applications"},{"displayLabel":"TASK_OUT_ACCOUNT_GROUP_AGGREGATION_TOTAL","attributeName":"total"},{"displayLabel":"TASK_OUT_ACCOUNT_GROUP_AGGREGATION_CREATED","attributeName":"groupsCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_GROUP_AGGREGATION_UPDATED","attributeName":"groupsUpdated"},{"displayLabel":"TASK_OUT_ACCOUNT_GROUP_AGGREGATION_DELETED","attributeName":"groupsDeleted"}],"attributes":{"creatorRequestId":"994096fc72d94e5fb887fff317f95068","creatorStack":"tpe","taskStartDelay":null,"qpocJobId":"d80cbd92-d638-4057-a528-d9d7d47719a3","creatorSpanId":"e89f44eaeaa24fb991c0f00e14f12410"}}
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: Aggregated successfully.
Complete.
------------------------------------------------------------------------------------------------------------
 Elapsed time:       1 seconds                      
 Files processed:    1                              
------------------------------------------------------------------------------------------------------------
 Success:            1                              
	/Users/neil.mcglennon/test/resources/entitlements/f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv
------------------------------------------------------------------------------------------------------------
 Error:              0                              
------------------------------------------------------------------------------------------------------------
 Skipped:            0                              
------------------------------------------------------------------------------------------------------------

Additional Details

This section outlines additional details related to troubleshooting, frequently asked questions, and other technical details.

File Naming Convention

The File Upload Utility analyzes at the files or directories you specify, and then looks for the application’s source ID in the beginning of the file name of the file. For example, for a file named 2c918087701c40cf01701dfdf2c61e2a-AuthEmployees.csv a source ID 2c918087701c40cf01701dfdf2c61e2a would be returned. A source ID is required for either account or entitlement aggregation to succeed as that is what the REST APIs require. To find the source ID, you can see this in the browser’s URL in your browser, or via source REST APIs.

If File Upload Utility doesn’t find the source ID in the file name, then it will skip the file and move on to the next. Everything will be logged as output, so you are able to determine which file(s) were processed or not. To test and see if your source ID can be found, run the simple regeular expression on the file name:

^(\s)?([0-9a-f]{32})

File Naming Backward Compatibility with Older Source IDs

For those who are upgrading from previous versions of File Upload Utility, you’ll know that older source IDs (e.g. 184744) are much shorter than newer source IDs (e.g. 2c918087701c40cf01701dfdf2c61e2a). In an attempt to keep backwards compatibility, if File Upload Utility detects an older short source ID (e.g. 184744), it will attempt to look up which new source ID this might map to. In order to do this, File Upload Utility will iterate through all sources in the system and download a list. This is visible when running File Upload Utility with the --verbose option:

...
Analyzing directory: /Users/neil.mcglennon/test/resources
Analyzing account file: 184744-AuthEmployees.csv
        File [184744-AuthEmployees.csv]: Detected with older source ID reference [184744]. Attempting to resolve.
------------------------------------------------------------------------------------------------------------
 Building source file lookup table. Starting source iteration. 
 To avoid this scan, please switch to new Source IDs in your file names.
 Older Source ID references will not be used in the future.
------------------------------------------------------------------------------------------------------------
 184744     : 2c918087701c40cf01701dfdf2c61e2a      
 226454     : 2c9180877a3f5acc017a43c671103c03      
 228616     : 2c9180877c337e84017c5261e23f1ec4      
 236466     : 2c918087825f9081018265298ff14eee      
 237117     : 2c91808783c7d7fd0183c882da42115d      
 237118     : 2c91808783c7d8070183c88371a3112a      
 215594     : 2c9180887769630b01776ee9be6f587d      
 237091     : 2c918088838b1cce0183c2b71ef23f05      
 228795     : 2c9180897cb94778017cc3a4af3e3d0e        
------------------------------------------------------------------------------------------------------------
        File [184744-AuthEmployees.csv]: Successfully resolved old source ID reference [184744] to new source ID reference [2c918087701c40cf01701dfdf2c61e2a]
        File [184744-AuthEmployees.csv]: Submitting Account Aggregation: Source ID[2c918087701c40cf01701dfdf2c61e2a], Disable Optimization[false]
        File [184744-AuthEmployees.csv]: Aggregation response: {"success":true,"task":{"type":"QUARTZ","id":"af1b82086c194f89acce82e156ff6e61","name":"Cloud Account Aggregation","description":"Aggregates from the specified application.","parentName":null,"launcher":"System","created":1714574212775,"launched":1714574212788,"completed":null,"completionStatus":null,"messages":[],"returns":[{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_APPLICATIONS","attributeName":"applications"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_TOTAL","attributeName":"total"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_OPTIMIZED","attributeName":"optimizedAggregation"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_IGNORED","attributeName":"ignored"},{"displayLabel":"TASK_OUT_UNCHANGED_ACCOUNTS","attributeName":"optimized"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_CREATED","attributeName":"created"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_UPDATED","attributeName":"updated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_DELETED","attributeName":"deleted"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_MANAGER_CHANGES","attributeName":"managerChanges"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_BUSINESS_ROLE_CHANGES","attributeName":"detectedRoleChanges"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_EXCEPTION_CHANGES","attributeName":"exceptionChanges"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_POLICIES","attributeName":"policies"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_POLICY_VIOLATIONS","attributeName":"policyViolations"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_POLICY_NOTIFICATIONS","attributeName":"policyNotifications"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SCORES_CHANGED","attributeName":"scoresChanged"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SNAPSHOTS_CREATED","attributeName":"snapshotsCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SCOPES_CREATED","attributeName":"scopesCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SCOPES_CORRELATED","attributeName":"scopesCorrelated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SCOPES_SELECTED","attributeName":"scopesSelected"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_SCOPES_DORMANT","attributeName":"scopesDormant"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_UNSCOPED_IDENTITIES","attributeName":"unscopedIdentities"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_CERTIFICATIONS_CREATED","attributeName":"certificationsCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_CERTIFICATIONS_DELETED","attributeName":"certificationsDeleted"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_APPLICATIONS_GENERATED","attributeName":"applicationsGenerated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_MANAGED_ATTRIBUTES_PROMOTED","attributeName":"managedAttributesCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_MANAGED_ATTRIBUTES_PROMOTED_BY_APP","attributeName":"managedAttributesCreatedByApplication"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_IDENTITYENTITLEMENTS_CREATED","attributeName":"identityEntitlementsCreated"},{"displayLabel":"TASK_OUT_ACCOUNT_AGGREGATION_GROUPS_CREATED","attributeName":"groupsCreated"}],"attributes":{"eventId":4311625,"appId":"2c918087701c40cf01701dfdf2c61e2a","optimizedAggregation":"enabled"},"progress":null}}
        File [184744-AuthEmployees.csv]: Aggregated successfully.

For files that are not able to resolve, with the --verbose option enabled, you should see messages like this:

Analyzing account file: 82343-accounts.csv
        File [82343-accounts.csv]: Detected with older source ID reference [82343]. Attempting to resolve.
        File [82343-accounts.csv]: Unable to resolve old source ID reference [82343] to new source ID reference. This file will be skipped.
        File [82343-accounts.csv]: Does not contain a valid source ID. Skipping...

This source file lookup table is generated once per execution of File Upload Utility and kept in memory. For organizations with a lot of sources, this might be somewhat expensive, and sources going forward might not have references to old source IDs going forward. To prevent this iteration, rename the files with the more modern file namings standard as mentioned in File Naming Convention.

Multiple Files or Directories

File Upload Utility does allow specifying multiple files or directories, by supplying the --file or -f options multiple times.

Here is example usage:

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/file1/ --file /Users/neil.mcglennon/test/file2/

Environment Variable Configurations

File Upload Utility does allow usage of environmental variables for the client ID and secret configurations. By providing option --clientId env the client ID will be sourced from the SAIL_CLIENT_ID environment variable. By providing option --clientSecret env the client secret will be sourced from the SAIL_CLIENT_SECRET environment variable.

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId env --clientSecret env --file /Users/neil.mcglennon/test/resources/ -R
------------------------------------------------------------------------------------------------------------
 SailPoint File Upload Utility
------------------------------------------------------------------------------------------------------------
 Version:            4.1.0                          
 Date:               2024-09-10 16:15 CST           
 Docs:               https://developer.sailpoint.com/discuss/t/file-upload-utility/18181 
------------------------------------------------------------------------------------------------------------
 --clientId derived from $SAIL_CLIENT_ID
 --clientSecret derived from $SAIL_CLIENT_SECRET
 URL:                https://example.api.identitynow.com 
 Client ID:          d0b28ce1b2694b64949dd546dd1ff574 
 Files:              /Users/neil.mcglennon/test/resources
 ObjectType:         account                        
 Optimization:       true                           
 Recursive:          true                           
 Extensions:         csv                            
 Simulation:         false                          
 Verbose:            true                           
 Timeout:            10000                          
------------------------------------------------------------------------------------------------------------
Checking credentials...
Analyzing directory: /Users/neil.mcglennon/test/resources
Analyzing account file: 184744-AuthEmployees.csv
...

Interactive Password Entry

File Upload Utility does allow secrets to be entered in an interactive fashion - for the --clientSecret and the --proxyPassword.

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546dd1ff574 --clientSecret --file /Users/neil.mcglennon/test/resources/ -R
Enter value for --clientSecret (SailPoint Client Secret (PAT)): a34xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1df
------------------------------------------------------------------------------------------------------------
 SailPoint File Upload Utility
------------------------------------------------------------------------------------------------------------
 Version:            4.1.0                          
 Date:               2024-09-10 16:15 CST           
 Docs:               https://developer.sailpoint.com/discuss/t/file-upload-utility/18181 
------------------------------------------------------------------------------------------------------------
 URL:                https://neil-test.api.identitynow.com 
 Client ID:          d0b28ce1b2694b64949dd546dd1ff574 
 Files:              /Users/neil.mcglennon/test/resources 
 ObjectType:         account                        
 Optimization:       true                           
 Recursive:          true                           
 Extensions:         csv                            
 Simulation:         false                          
 Verbose:            true                           
 Timeout:            10000                          
------------------------------------------------------------------------------------------------------------
Analyzing directory: /Users/neil.mcglennon/test/resources
Analyzing account file: 184744-AuthEmployees.csv
...

Proxy Configurations

Web (HTTP) proxies are supported by supplying proxy settings with the --proxyHost and --proxyPort options. Here is example usage:

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/resources/ --proxyHost 192.168.10.1 --proxyPort 8080

For authenticated proxies, add --proxyUser and --proxyPassword options as well. Here is example usage:

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/resources/ --proxyHost 192.168.10.1 --proxyPort 8080 -proxyUser MyProxyUser --proxyPassword pr0xyP@s$w0rD

Timeout Configurations

Timeout settings are configurable by supplying a number of milliseconds to configure the timeout, using the --timeout, or -t options respectively. The default timeout in the File Upload Utility is 10 seconds (10,000 ms).

$ java -jar sailpoint-file-upload-utility.jar --url https://example.api.identitynow.com --clientId d0b28ce1b2694b64949dd546de1ff574 --clientSecret a34...1df --file /Users/neil.mcglennon/test/resources/ --timeout 15000

Often timeouts are an indication that File Upload Utility is attempting to communicate with the SailPoint Cloud and not getting a response. Usually, this due to network security controls, such as firewalls, preventing the communication. Work with your network teams to make sure you can reach the SailPoint Cloud, and if necessary, adjust Timeout Configurations or even leverage Proxy Configurations if so required.

Troubleshooting

Timeouts

Timeouts are usually an indication that File Upload Utility is attempting to communicate with the SailPoint Cloud and not getting a response. Usually, this due to network security controls, such as firewalls, preventing the communication. Work with your network teams to make sure you can reach the SailPoint Cloud, and if necessary, adjust Timeout Configurations or even leverage Proxy Configurations if so required.

See also Timeout Configurations.

401 (Unauthorized) Errors

When File Upload Utility first connects to the SailPoint Cloud, it checks credentials to make sure the supplied Client ID and Client Secret are valid, and provide an access token. If this fails, you’ll typically see an error that looks like this:

Checking credentials...
Error Logging into Identity Security Cloud.  Please check your credentials and try again. [Error authenticating with credentials: 401 ]

In order to remedy this, make sure you are using a correct Personal Access Token on the right SailPoint tenant and URL.

403 (Forbidden) Aggregation Errors

When File Upload Utility first connects to the SailPoint Cloud, it checks credentials to make sure the supplied Client ID and Client Secret are valid, and provide an access token. While an access token may be valid, the token’s usage may not have adequate access rights in order to carry out account or entitlement aggregations. Make sure that the Personal Access Token supplied has correct access rights needed. At current time of writing ORG_ADMIN, SOURCE_ADMIN or SUB_SOURCE_ADMIN access rights are required.

400 (Bad Request) Aggregation Errors

File Upload Utility doesn’t perform any sort of validation or manipulation of files; it merely acts as transport to the SailPoint Cloud via the account aggregation and entitlement aggregation REST APIs. Once the files are received, the SailPoint Cloud can provide errors back vis-a-vis the REST API responses. These generally look like these:

Analyzing account file: f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv
	File [f1f3b747be924745afbc0c8a53f71baf-file-test-entitlement-feed.csv]: Error: HTTP Code[400] Message[] Body[{"detailCode":"400.1.3 Illegal value","trackingId":"44a4f5a299d540df84537fc51fd9271f","messages":[{"locale":"und","localeOrigin":"REQUEST","text":"Column: \"[displayName, created, description, modified, entitlements, permissions]\" is unknown and/or column: \"[givenName, familyName, e-mail, location, manager]\" is missing"},{"locale":"en-US","localeOrigin":"DEFAULT","text":"Column: \"[displayName, created, description, modified, entitlements, permissions]\" is unknown and/or column: \"[givenName, familyName, e-mail, location, manager]\" is missing"}],"causes":[]}]

If any sort of 400 Bad Request errors are given, check your file configuration is relevant for the type of aggregation you are doing. Sometimes this can be due to not specifying the correct object class on entitlement aggregations; this can be easily remedied by checking the --objectType option, and making sure the corresponding entitlement schema exists.

500 (Internal Server Error) Aggregation Errors

If you see a 500 Internal Server Error, this means there is a problem with the SailPoint Cloud processing itself. If you see this, it is best to reproduce the issue using the REST APIs and alert SailPoint Support.

Frequently Asked Questions

• Q: Can I configure values with environment variables?

  • A: By providing --clientId env the client ID will be sourced from the SAIL_CLIENT_ID environment variable. By providing --clientSecret env the client secret will be sourced from the SAIL_CLIENT_SECRET environment variable.

• Q: How can I secure my --clientSecret values?

  • A: The --clientSecret value can be entered via arguments, and provided and secured by any other scripts calling File Upload Utility. Additionally, File Upload Utility 4.1.0 supports interactive entry by providing --clientSecret with no value. Client secrets can be provided from the SAIL_CLIENT_SECRET environment variable by configuring the command line option as --clientSecret env.

• Q: What are you doing about CC/V2 APIs being deprecated?

  • A: As of File Upload Utility 4.0, this leverages supported V3 SailPoint REST APIs. No older or private (CC) REST APIs are used in this.

• Q: Does File Upload Utility still support older (CC) Source IDs?

  • A: See File Naming Backward Compatibility with Older Source IDs

• Q: How is File Upload Utility’s transmission secured?

  • A: File Upload Utility uses SailPoint REST APIs which are all secured over HTTPS / TLS 1.2.

• Q: Does File Upload Utility support file-level encryption?

  • A: File Upload Utility does not currently support encryption or decryption of files themselves. However, additional scripting could surround File Upload Utility which could encrypt, decrypt, modify, or move files.

• Q: Does File Upload support reading from SFTP, FTP, or SSH locations?

  • A: File Upload Utility only supports reading from local drives, not from remote SFTP, FTP, or SSH locations. However, additional scripting could surround File Upload Utility which could connect to SFTP, FTP, or SSH sessions to transmit, modify, or move files.

• Q: What REST APIs does this use?

  • A: We use the following REST APIs:
  • Account Aggregation - For account aggregations
  • Entitlement Aggregation - For entitlement aggregations
  • List Sources - For source lookup
  • Get Source - For source lookup

Upgrading from Previous Versions

If you are coming from previous versions of File Upload Utility, you’ll notice a few important changes:

• Java Requirements - File Upload Utility 4.0.0 requires Java JDK 11+
• Command Line Options - Some command line options have changed. See Options for details.
• SailPoint APIs - We are no longer using older (deprecated) CC Source APIs, and instead use newer Source APIs for Account Aggregation and Entitlement Aggregation.
• Source ID Format - We use newer source ID (e.g. 2c918087701c40cf01701dfdf2c61e2a) formats in file names. If you have older source ID (e.g. 184744) formats we’ll attempt to look it up. See File Naming Backward Compatibility with Older Source IDs
• Proxy Type - For those using proxies, proxy type no longer needs to be given.

[jrossicare (Jason Ross)]

Hi @neil_mcglennon
Is the idea to move away from the Short ID for source name, once /cc/ API’s are deprecated?

[Dharani_01 (Bhavadharani Arulsivaraman)]

Hi Everyone,

While following this process and executing this command - java -jar FileUploadUtility.jar -url https//****.api.identitynow.com -client_id *** client_secret **** -entitlement “path of file”, facing this below error.

• “Exception in thread “main” com.beust.jcommander.ParameterException: Was passed main parameter ‘-entitlement’ but no main parameter was defined in your arg class”

• Exception in thread “main” com.beust.jcommander.ParameterException: Was passed main parameter ‘-e’ but no main parameter was defined in your arg class.

Thanks in Advance.

Regards,
Dharani

[chandramohans27 (Chandra Mohan S)]

@neil_mcglennon

I’m also facing the same error.

• Exception in thread “main” com.beust.jcommander.ParameterException: Was passed main parameter ‘-entitlement’ but no main parameter was defined in your arg class”
• Exception in thread “main” com.beust.jcommander.ParameterException: Was passed main parameter ‘-e’ but no main parameter was defined in your arg class.

I’m using File Upload Utility version 3.0.2, but it looks -e option is not there in this version.

[E5706819 (Matt Manalo)]

Hello,

We’re receiving an Error Code 403 when trying to use the FileUploadUtility. I’ve read that if upload have failed, it will only provide generic error. Can you please advise what could possibly gone wrong on our command?

java -jar FileUploadUtility.jar -url -client_id -client_secret -f <“file path”> -R -v -timeout 90000

[colin_mckibben]

A 403 means the access token you provided does not have sufficient privilege to execute the API calls. The API calls in question will need a personal access token with ORG_ADMIN capabilities and sp:scopes:all scope to work properly. Here is the guide on generating a personal access token.

Authentication | SailPoint Developer Community

When you are prompted to apply scopes, make sure sp:scopes:all is selected. You will be given a client ID and client secret, which you will use when invoking the JAR file.

[mhoang (Minh Hoang)]

Hi,

We are using the File Upload Utility but an older version v2.0.3. Will this also be affected with the Decommission of Non-public APIs?

Thanks.

[colin_mckibben]

Yes, this utility uses the private APIs that are going away March 31st. Once the replacements are available, this utility will be updated.

[neil_mcglennon]

We are currently waiting on the new V3 Aggregation APIs, however once these APIs arrive, we plan on moving to the V3 source ID syntax in the next version of File Upload Utility.

For compatibility reasons, if a V3 source ID is not detected, we are also going to attempt to provide a source lookup on the short ID to get a V3 source ID. This is meant to help ease the transition for customers who may be unaware.

More details to come once we have definitive APIs available.

[neil_mcglennon]

The -e option was introduced in the later versions of File Upload Utility. It is there for version 3.0.5. Give that a try!

[neil_mcglennon]

For anyone wondering about questions regarding File Upload Utility and CC API Deprecations, see this related post:

File Upload Utility Plans with API Deprecations

Hello Developer Community, I am the author and owner of the File Upload Utility, which I know many of you rely on for source aggregations on a daily basis. I just wanted to provide you all with a quick status update on future plans, especially in light of CC API Deprecations announced, to help clarify, set expectations, and answer any lingering doubts. Here is a quick rundown of questions we’ve received and answered to date: Q: Is the File Upload Utility still an active tool? A: Absolut…

[udayputta]

hi @neil_mcglennon
In the current version I observed that after the execution is completed the pointer does not come out unless I press Ctrl+C. Wanted to check if that will be fixed in next version and pointer will not stay there until someone press Ctrl+C. This would be helpful when the jobs are scheduled using task schedular.

FYI. I have just used the simple commands without any other options

java -jar “C:\SailPoint\FileUploadUtility\FileUploadUtility.jar” -url https://.api.identitynow.com -client_id e5fc9…8fb1 -client_secret 7d…89e -f C:\ftpdat\20148-UserAccountFeed.csv

[ronaldchuiASB (Ronald Chui)]

hi @neil_mcglennon

Where can we download the 3.0.5 jar file? The link in SailPoint is still pointing to 3.0.2.

Thanks, Ronald

[stellaantony (Stella Mary)]

Hi @neil_mcglennon
we use file upload utility to push feed file from our server to cloud. It fails with below error when we execute the jar file. we created client id and secret id twice to sort out this session issue. but always it fails with below error.

could you please help us to resolve this error.

Thanks
Stella

[neil_mcglennon]

Hi @Stella - Can you share more about how you are invoking this? The screenshot is cut-off and this doesn’t look to be the typical CLI type of invocation.

Feel free to DM me directly, or drop me an email if you’d rather not post publicly.

[adebomol2024 (adebolu omololu)]

Team, we are getting generic 503 error codes using the file upload util. We are using a version from 2021. 503 means IDN was unavailable at these times. We have been seeing 503 error messages for the past 3 weeks. We just discovered based on a workflow issue that is configured to write back business email to HR source. Please advice on what the 503 could be as a result of.

[dernc (David Ernce)]

@adebomol2024

503 means IDN was unavailable at these times. We have been seeing 503 error messages for the past 3 weeks. We just discovered based on a workflow issue that is configured to write back business email to HR source. Please advice on what the 503 could be as a result of

We’ve been getting 503 errors as well, starting around Feb 6 of this year. The Ops team has to manually check for errors and upload files through the UI. Interesting that it hasn’t been reported until now. And that there isn’t a list of known errors. The error messages themselves are pretty generic.

Analyzing file: Filename-yadda-yadda.csv
*** File [Filename-yadda-yadda.csv] had an error: Code[503] Message***

[kbeckwith (Kyle Beckwith)]

Why don’t you build these functions into the UX of the product. Seems silly you need to make a tool like this.

[dernc (David Ernce)]

@neil_mcglennon

https://community.sailpoint.com/docs/DOC-3140

What’s the current version of the utility? The download page here provides 3.0.2, which is the same as we’re currently using and is from October 2021. But the usage example shown on this page shows version 3.0.5 from 2022-12-01. Many of us are obviously waiting for the next version which will use the new (but really late) API endpoints. But since we get sporadic errors on 3.0.2, am wondering if a later version is available.

[neil_mcglennon]

Why don’t you build these functions into the UX of the product. Seems silly you need to make a tool like this.

You don’t need this tool. You can upload your files straight from the user interface if you wish.

The File Upload Utility uses those same APIs that the user interface uses, but as a way to bulk upload many files automatically from a remote file system. People use this tool often in conjunction with file handling utilities.