New Capability: Delimited File Connector - New VA based connectivity and Enhancements

Announcements Product News
, ,
1

New Capability

We are pleased to announce enhancements to the SailPoint Delimited File Connector, which now includes an option for VA-based connectivity. This new approach supports various file transport mechanisms for reading files from SFTP, FTPS, SCP servers, and Amazon S3 locations. It also includes support for encryption, filtering, parsing, and data merging capabilities.

You now have more flexibility when configuring Account and Group Schemas, including the ability to set up multiple group object schemas.

Simplified File Access: On-Premise and Cloud

  • You can easily connect to your SFTP, FTPS, and SCP servers to retrieve files. By providing the necessary connection settings, including the host, port, username, and password for your server, you gain the flexibility to either get the file on-demand or schedule the aggregation for a later time.
  • For those utilizing AWS, retrieving files from your S3 location is just as straightforward. Simply provide your S3 bucket name and region, and authenticate using your IAM user credentials to access your files seamlessly.

By offering these versatile connection options, we ensure that your data is always within reach, regardless of where it’s stored.

Mastering Your Data Files: Flexible Configuration

  • Effectively managing and processing data files is a cornerstone of robust data integration. Getting your data parsed, cleaned, and organized correctly is the key.

Step 1: File Parsing

You have two great options for file parsing:

  • Delimited: This is your go-to for simple, structured files. Is your data separated by commas, tabs, or another special character? Just tell the connector what that delimiter is, and it will handle the rest. You can even use unicode characters (like \\u0009 for a tab) for extra precision!
  • Regular Expression: For data that’s a bit more wild and less structured, a Regular Expression is your best friend. This powerful cofnigration lets you define a pattern to find and extract the exact pieces of data you need. You’ll just need to name the columns you’re creating so everything stays organized.

A quick note on columns: If you’re using a Regular Expression, defining your columns is a must. If you’re going the Delimited route, you only need to worry about this if your file doesn’t have a header row or if you want to give the columns friendlier names to use later on.

Step 2: Filtering

Now that you’re reading the file, it’s time to clean it up. You don’t want junk data cluttering your results, right? Filtering helps you focus on what’s important.

  • Skip the Intro: Got a few header lines or introductory text at the top of your file? Simply tell the connector how many lines to skip, and it will jump right to the good stuff.
  • Remove the Empties: Sometimes, you’ll get lines that parse but don’t actually contain any useful data. Just enable the Filter Empty option to toss them out automatically.
  • Ignore Comments: Many data files include comment lines for context. You can specify a comment character (like ‘#’), and any line starting with it will be completely ignored.
  • Be Specific with String Filters: This is where you can get really precise. Want to exclude every record from the “Manufacturing” department? A simple Filter String like department == /"Manufacturing/" will do the trick, ensuring only the most relevant data makes it through.

Step 3: Data Merging

What if you have related data spread across multiple rows? That’s where merging comes in. This feature lets you combine information for a single account from different lines in your file. For the best results, your data should be sorted by a unique identifier. But don’t worry if it’s not - the connector can build a sorted version in memory to handle the merge. Plus, you can even choose to “Ignore case while merging” to make sure “UserA” and “usera” are treated as the same.

By mastering these parsing, filtering, and merging settings, you’ll gain incredible flexibility and ensure your data is always clean, accurate, and ready for action.

File Encryption

Data security is paramount, and sometimes that means your files arrive under lock and key. If you’re working with files that have been encrypted using PGP (Pretty Good Privacy), we’ve got you covered.

Simply enable the PGP Encrypted setting. You’ll then be prompted to provide your PGP private key and the password that goes with it. Once you’ve entered those credentials, the connector will automatically decrypt the file’s data, making it ready for processing without compromising on security. It’s a seamless way to handle sensitive information securely.

Note: You can continue to use the existing Delimited File Connector (SaaS) option and there are additional enhancements in the “File Settings” to provide better flexibilities while reading the file.

Documentation:

Release Details

  • Identity Security Cloud - Available.
4 Likes
New Delimited File functionality in Sandbox?
2

Much awaited feature!!! Of ISC

1 Like
3

Great quality of life improvement for delimited file sources! Would be nice to see s3 bucket authorization support an IAM Role added to the EC2 instance running the Virtual Appliance (similar to credential provider) instead of requiring the Access key ID and Secret Access Key in the config.

4

This sounds excellent!

I am keen to try this out.

1 Like
5

This is a great release. The existing delimited sources in the system seem to be defaulted to SAAS. Is there a way we can upgrade these from SAAS to VA for us to use the new connectivity functionality? Or would this have to go through re-creation? @dinesh_mishra

2 Likes
6

What about Azure storage?

7

Hi @Karthik_P, currently you need to create a new source for VA as the use cases are different. Thank you!

8

Hi @zeross, it’s in the future roadmap, no ETA at this point of time. We will share more updates as per the progress. Thanks!

9

Will there be support for SFTP using Public key any time soon? I do not see that option available currently.

1 Like
10

Hi @dinesh_mishra

Does the account schema definition need to follow the column order of the file?

I’m trying to confirm whether, if the column order in a file changes, the connector will still be able to read the data from that file.

Thx !

11

Switching to a new source also requires reset of anything using related entitlements (cert filters, tags, AP’s, workflows etc.) in order to use the new feature, so not being able to move to VA is unfortunate.

Will the ability to switch from existing delimited file source to VA source possibly be added at a later date?

12

Thank you for this new feature. It will help to eliminate a lot of manual processes which make running a delimited file source prone to error. We’ll also be able to see fresher data without requiring daily baby-sitting, which starts to look unscalable even after only a few sources.

13

Thank you for this new feature. It will be helpful

14

Thanks for sharing this—great update!
The new VA-based Delimited File Connector enhancements look very promising. I’ll be testing this in sandbox soon. Really appreciate the detailed explanation.

15

Hello, can we convert the existing one to the new connector with minimal changes?