✨ Community AI Upgraded to Claude 3 Opus ✨

[derek_putnam]

Starting today, Community AI has been upgraded from Anthropic’s Claude 3 Sonnet, to Claude 3 Opus!

Why the upgrade?

In this fast-changing world of AI, we’re constant looking at what we can do to improve your experience in our community, and with our products. As the AI industry moves and improves, we’ll do our best to follow it and create better community experience for you in kind!

Code Generation

Opus has a ~16% increase in quality of code generation over Sonnet, moving us closer to providing users with a better assistant in building on our platform. To learn more about the specifics behind Claude 3 training, read the full whitepaper here from Anthropic.

Vision Capabilities

Sometimes code isn’t in an ideal state. A screenshare? A PDF? Either way, we now offer vision capabilities in Community AI. Upload a screenshot of some code, and Community AI can help you understand what is going on!

Not sure what Community AI is?

In April, we announced Community AI as an assistant to users actively developing on the Identity Security Cloud. In particular, working directly with APIs, Workflows, SaaS Connector, Connector Customizers, Transforms, and Rules. You can read the initial release here:

Introducing Community AI

[Developer community AI announcement] Our team is thrilled to announce the release of Community AI, our community feature with the potential to transform your identity development! Community AI is an AI assistant to help you in developing identity solutions on the Identity Security Cloud. Checkout our announcement video from Developer Days 2024 to see our Community AI in action! What’s great about Innovation being one of SailPoint’s Four I’s is that it happens across the business—…

Today, Community AI is under active development, and is available to members of SailPoint’s community Ambassador Program. If you’re interested in becoming an ambassador and getting access to perks like your own private Identity Security Cloud tenant, early access to Community AI, and more…checkout the program, and we’d love to have you join us!

Announcing The Developer Community Ambassador Program

[ambassador-program-banner-ambassador] We consider it our responsibility to provide a place and an experience that facilitates not only our ability to support you, but your ability to support each other. To do this this, we’re announcing the Developer Community Ambassador Program. Some of the rewards for participating include: Exclusive IdentityNow Test Tenant Exclusive Access Recognition How do I get started? Just click right here @ambassadors_in_training to get started! …

Questions?

Please don’t hesitate to reach out with any questions or comments! If you need help here in the community let us know here, or you can send a message directly to our @community_managers!

[WyssAJ01 (Adam Wyss)]

Thanks for the update, Jordan! I’m glad that you guys are keeping the tool fresh, and I hope that it’s something that sticks around for a while as it really can help decrease time to production for both us as partners and for customer ambassadors.

[derek_putnam]

I’m really glad you’re enjoying it, and I appreciate all the feedback you’ve been sharing in your AI chats, too!

How often do you find yourself using Community AI? A few times a day? Once a week?

[Bakhari (Bakhari Burrell)]

Thanks for the update dude! I just messed around with GPT-4o yesterday and was floored by the leaps and bounds it is making. Claude 3 Opus and GPT-4o are like neck and neck in my opinion. But Claude’s integration into Sailpoint makes using for work configurations a lot easier because it understands Sailpoint and I don’t have to tell it what it needs to know first, like I used to have to do with GPT. Good stuff! Keep up the great work!

[derek_putnam]

I’m glad you’re enjoying it! What are some common tasks you’re using Community AI to help you with?

[Bakhari (Bakhari Burrell)]

I mostly use it to throw together quick transforms without having to go and dig into my code. I don’t write transforms that often, so the syntax doesn’t stick in my head. Having the AI throw it together helps me remember little details I miss (like adding the TYPE to the syntax) before I go to test.
I’ve also used it to help me with JDBC connectors, but that was more so helping me understand how ISC handles information exchange with JDBC connectors. It’s not as straightforward as I thought it would be, but AI has pointed out things I completely missed in my initial setup. (Like understanding that a connection setup is not needed in the JDBC rule code because it’s already being handled in the UI). Very interesting and very helpful! Makes it easy so I don’t have to scan through tons and tons of documentation. I just ask AI. XD

[derek_putnam]

Is there anything in the realm of SailPoint that you wished a conversational AI experience would offer you?

[AroraA3 (Amardeep Singh Arora)]

Is there a possibility that we would be able to use/see this for IIQ as well?

[derek_putnam]

Hey Amardeep, fair question! There are no plans today to enable this for IdentityIQ.

[AroraA3 (Amardeep Singh Arora)]

Would really love to see this for IIQ, is there somewhere we can request for this and see how many people would want this like a poll or something?

[derek_putnam]

Hey Amardeep! Right now, the best place to get visibility for something like this is right here in the community (so you’ve got a head start!). I would also recommend submitting an idea to product management via our Ideas Portal:

https://ideas.sailpoint.com/

[angelo_mekenkamp]

I occasionally give it a chance, sometimes on questions I already know the answer to (to test the AI) and sometimes on questions it does not know the answer to, but it hasn’t been able to give me correct answers yet on the questions I needed an answer to.

I think this is mainly because the AI is trained on the documentation and whenever the documentation is incomplete, it doesn’t have the knowledge to it. Downside it also that it will answer very confidently while basically guessing rather than saying “I don’t know the answer to this question, perhaps you can try if the following works…”.

One example I just tried is asking AI how I can get tomorrows or yesterdays date as a variable in workflows. For this you would need to get the current date and then add or subtract one day. There is no documentation for getting the current date (so the AI guesses something that does not work) and there is documentation on subtracting dates, where the documentation specifically mentions that workflows breaks if you try to generate a date that is before the current date, and the AI did not pick up this limitation and therefore suggests a solution that does not work.

When asking it to create a transform to concatenate two identity attributes it will give correct information, but then describing the issue takes more time than building it myself using the VS code plugin.

All in all I experience that when it comes to ISC development related questions, I spend more time talking to the AI than what I save in return.

Every now and then I will go back to the AI again to see if it has improved though, but currently I would recommend reading the documentation (training oneself on real SailPoint statements) rather than asking the AI questions (which taints your training by overly confident wrong answers).

Kind regards,
Angelo

[AroraA3 (Amardeep Singh Arora)]

Hey Jordan! I just submitted the idea as suggested, would love to see this in the future!

[Bakhari (Bakhari Burrell)]

@derek_putnam ! THIS! This is essentially what I was going to send to you, but Angelo has done a fantastic job of condensing the word vomit I had been working on. Thanks Angelo!

[tysremi (Remi Tys)]

Just wanted to provided some feedback as well.
I don’t know all possible API’s that are available by heart, and the search in the developer portal is quite lacking (It’s quite hard to find the correct API), so since recently I try to ask which API to use or how to do a certain thing.

Oftentimes it goes well and it gives me some more info that I can get started with.
But I also had times where it completely invents things that it seems quite certain about.
Like this conversation:

It seems to produce a valid answer to my question, but as I was not finding this specific API I asked where to find the documentation and it linked me to a non-existing URL (which also seems like it might go somehere…)
https://developer.sailpoint.com/idn/api/v3/get-historical-exceptions

Either it’s ahead of it’s time and has access to unpublished, future API’s or there’s some hallucination going on…

[derek_putnam]

@angelo_mekenkamp

I think this is mainly because the AI is trained on the documentation and whenever the documentation is incomplete, it doesn’t have the knowledge to it.

This is exactly right! We’re constantly working on improving our docs as a result of this, which of course is benefiting both our users directly as well as improving the AI. It is also trained only on docs on developer.sailpoint.com today, but we would like to extend that.

@angelo_mekenkamp

Downside it also that it will answer very confidently while basically guessing rather than saying “I don’t know the answer to this question, perhaps you can try if the following works…”.

Yes, this is the unfortunate side effect of hallucinations of LLMs in AI, and I expect these foundational models will get better over time. We will also continue to improve our process based on your feedback here!

[derek_putnam]

Remi, great to see you! Thanks so much for this feedback—incredibly detailed!

@tysremi

search in the developer portal is quite lacking (It’s quite hard to find the correct API)

Could I reach out to you on this? I’d love to learn more? As you surmised, there is some hallucinating going on and something that we will continue to improve as ambassadors like you continue to help with!

[vishal_kejriwal1 (Vishal Kejriwal)]

This is awesome @derek_putnam !!