Handling non-Latin / special characters in account name & email transforms

Identity Security Cloud (ISC) ISC Discussion and Questions
1

Hi team,

We generate sAMAccountName, UPN, and email from HR name fields using a transform chain (concat → lower → decomposeDiacriticalMarks → replace [^a-zA-Z]). This handles standard Latin accents well (é→e, ñ→n). We’ve added a small replaceAll table on top to transliterate non-Latin Latin-script letters (Þ→th, ø→o, ß→ss, etc.) since decompose drops those.

Two questions:

Is a supplementary replaceAll table the recommended approach for these characters, or is there a cleaner/more maintainable pattern?
For fully non-Latin scripts (Cyrillic, Arabic, CJK) that can’t be transliterated by rule — is there a way for a workflow to detect that a name contained unhandled characters? Our understanding is that by the time the transform runs, the characters are already stripped, so the workflow only sees the cleaned result (e.g. an empty/short account name), not the original input. Is that correct, or is there a way to flag the original?

Thanks,

Sreesha

2

  1. Yes, a supplementary replaceAll table is currently the most practical approach for characters that are not handled by decomposeDiacriticalMarks (such as ß, Þ, Ø, Æ, Œ, etc.). Many customers maintain a transliteration map before applying the final regex cleanup. There isn’t an out-of-the-box ISC transform that performs full Unicode transliteration across all scripts.

  2. For Cyrillic, Arabic, CJK, and other non-Latin scripts, you should evaluate the original HR attributes before the sanitizing transform runs. Once your transform chain removes unsupported characters, downstream consumers only see the transformed value. If the result becomes empty or significantly shortened, the workflow cannot reliably determine what original characters were removed unless the original source attributes are preserved separately.

A common pattern is:

  • Keep the raw HR attributes (firstname, lastname) unchanged.

  • Create a separate generated attribute for sAMAccountName/UPN/email.

  • Use an Identity Attribute or workflow check against the original HR values to detect non-Latin characters with a regex.

  • Set a flag such as requiresManualNamingReview=true when unsupported characters are detected.

  • Original Name: Дмитрий Иванов

    Generated Account: “”

If you only inspect the generated account name, you cannot distinguish between:

  • Unsupported characters removed
  • Missing source data
  • Transform failure

Therefore, detection should occur against the original source attributes, not the final sanitized output.

3

Hi @SreeshaC93
Q1: replaceAll table vs cleaner pattern

The supplementary replaceAll table is actually the recommended approach for non-decomposable characters like Þ→th, ß→ss, ø→o in ISC. There’s no built-in transliteration transform, so a chained replaceAll lookup table before your decomposeDiacriticalMarks step is the cleanest maintainable pattern available today. Keep it as the first step in your chain so decompose handles the rest downstream.

Q2: Detecting unhandled non-Latin characters (Cyrillic, Arabic, CJK)

Your understanding is correct — by the time the transform completes, the original input is gone and the workflow only sees the cleaned/stripped output. However, there is a way to flag it:

  • Use a separate detection transform on the raw HR name field (before your main chain) that applies a contains or matches regex like [^\x00-\x7F] to detect non-ASCII characters
  • Surface this as a separate identity attribute (e.g., hasNonLatinName = true)
  • Then trigger a workflow on Identity Change watching that flag → send alert/notification to HR team

This way the original value is evaluated before stripping, and your workflow has something actionable to react to.

Thanks!

4

Hello Sreesha. Welcome to the community.

A replaceAll mapping is the practical approach for characters that decomposeDiacriticalMarks does not convert, such as ß, Þ, and ø. For maintainability, keep the mapping in one named transform and reuse it through a reference transform wherever needed.

For detection, contains and matches are not standalone ISC transform types, and [^\x00-\x7F] would also flag accented Latin characters your chain already handles.

Instead, create a parallel detection transform that:

  1. Reads the raw HR name with a null-safe fallback.
  2. Applies the same transliteration and decomposeDiacriticalMarks steps.
  3. Strips all characters your naming standard allows (Latin letters, spaces, hyphens, apostrophes).
  4. Anything remaining is outside the configured naming standard.

A static transform returns the flag:

#if($unhandled == '')false#{else}true#end

Map this to an identity attribute like requiresManualNamingReview. Use an Identity Attributes Changed workflow filtered with:

$.changes[?(@.attribute == "requiresManualNamingReview" && @.newValue == "true")]

Add an Identity Created workflow as well to catch new identities whose flag is "true" at creation.

The raw HR value stays on the source account if it is in the source schema and aggregated, but is not automatically included in the event payload.

5

Thank you — my key takeaway from your answer is the shift to detecting against the original source attributes rather than the stripped output. That reframes the whole problem correctly. The point that an empty result is ambiguous (removed characters vs. missing data vs. transform failure) is also well taken — I’ll evaluate the raw name, and use the requiresManualNamingReview flag pattern you described. Appreciate it.

6

Thanks Gopi — the takeaway I’m keeping from your response is flagging on the raw HR field before the main chain runs. And a useful caveat I’ll act on: a broad non-ASCII match like [^\x00-\x7F] would also catch the accented Latin characters my chain already handles, so I’ll scope the detection to flag only what’s genuinely unhandled rather than everything non-ASCII. Appreciate the workflow-on-change pointer.

7

Hi Harish, thank you for detailed solution. My takeaway is the parallel detection transform: re-run the same transliteration + decompose, strip everything the naming standard allows, and flag whatever’s left via a static transform.

One follow up ask , since you laid out the workflow triggers: for the Identity created trigger, is the calculated flag reliably populated at the point the trigger evaluates? my concern is the attribute being computed as part of/after identity creation, so the workflow might fire before the flag is set — is Identity Attribute Changed the safer trigger here, or does Identity Created reliably see the freshly calculated value? Appreciate your help!

8

Hello Sreesha. According to SailPoint’s doc, the Identity Created payload contains all identity attributes configured in the identity profile. Since this flag is calculated from the same authoritative HR account, it should be populated when the trigger evaluates, assuming the transform completes successfully.

You can filter it with:

$[?($.attributes.requiresManualNamingReview == "true")]

I would use:

  • Identity Created for new identities whose flag is "true" at creation.
  • Identity Attributes Changed for later changes where the flag becomes "true".

So Identity Attributes Changed is not necessarily safer for initial creation. Identity Created is the documented trigger for that case. I would still test it with a new identity in your tenant. If the payload unexpectedly contains null attributes, use an unfiltered Identity Created trigger followed by a one-minute Wait and Get Identity step.

9

Thanks — this resolves my concern. I’ll test the detection workflow against a new identity in the tenant as you suggested. Appreciated.