✨ Community AI Upgraded to Claude 3 Opus ✨

Starting today, Community AI has been upgraded from Anthropic’s Claude 3 Sonnet, to Claude 3 Opus! :tada:

Why the upgrade?

In this fast-changing world of AI, we’re constant looking at what we can do to improve your experience in our community, and with our products. As the AI industry moves and improves, we’ll do our best to follow it and create better community experience for you in kind!

Code Generation

Opus has a ~16% increase in quality of code generation over Sonnet, moving us closer to providing users with a better assistant in building on our platform. To learn more about the specifics behind Claude 3 training, read the full whitepaper here from Anthropic.

Vision Capabilities

Sometimes code isn’t in an ideal state. A screenshare? A PDF? Either way, we now offer vision capabilities in Community AI. Upload a screenshot of some code, and Community AI can help you understand what is going on!

Not sure what Community AI is?

In April, we announced Community AI as an assistant to users actively developing on the Identity Security Cloud. In particular, working directly with APIs, Workflows, SaaS Connector, Connector Customizers, Transforms, and Rules. You can read the initial release here:

Today, Community AI is under active development, and is available to members of SailPoint’s community Ambassador Program. If you’re interested in becoming an ambassador and getting access to perks like your own private Identity Security Cloud tenant, early access to Community AI, and more…checkout the program, and we’d love to have you join us!

Questions?

Please don’t hesitate to reach out with any questions or comments! If you need help here in the community let us know here, or you can send a message directly to our @community_managers!

14 Likes

Thanks for the update, Jordan! I’m glad that you guys are keeping the tool fresh, and I hope that it’s something that sticks around for a while as it really can help decrease time to production for both us as partners and for customer ambassadors.

2 Likes

I’m really glad you’re enjoying it, and I appreciate all the feedback you’ve been sharing in your AI chats, too!

How often do you find yourself using Community AI? A few times a day? Once a week?

Thanks for the update dude! :grin: I just messed around with GPT-4o yesterday and was floored by the leaps and bounds it is making. Claude 3 Opus and GPT-4o are like neck and neck in my opinion. But Claude’s integration into Sailpoint makes using for work configurations a lot easier because it understands Sailpoint and I don’t have to tell it what it needs to know first, like I used to have to do with GPT. Good stuff! Keep up the great work!

1 Like

I’m glad you’re enjoying it! What are some common tasks you’re using Community AI to help you with?

I mostly use it to throw together quick transforms without having to go and dig into my code. I don’t write transforms that often, so the syntax doesn’t stick in my head. Having the AI throw it together helps me remember little details I miss (like adding the TYPE to the syntax) before I go to test.
I’ve also used it to help me with JDBC connectors, but that was more so helping me understand how ISC handles information exchange with JDBC connectors. It’s not as straightforward as I thought it would be, but AI has pointed out things I completely missed in my initial setup. (Like understanding that a connection setup is not needed in the JDBC rule code because it’s already being handled in the UI). Very interesting and very helpful! Makes it easy so I don’t have to scan through tons and tons of documentation. I just ask AI. XD

3 Likes

Is there anything in the realm of SailPoint that you wished a conversational AI experience would offer you?

Is there a possibility that we would be able to use/see this for IIQ as well?

3 Likes

Hey Amardeep, fair question! There are no plans today to enable this for IdentityIQ.

2 Likes

Would really love to see this for IIQ, is there somewhere we can request for this and see how many people would want this like a poll or something?

1 Like

Hey Amardeep! Right now, the best place to get visibility for something like this is right here in the community (so you’ve got a head start!). I would also recommend submitting an idea to product management via our Ideas Portal:

https://ideas.sailpoint.com/

1 Like

I occasionally give it a chance, sometimes on questions I already know the answer to (to test the AI) and sometimes on questions it does not know the answer to, but it hasn’t been able to give me correct answers yet on the questions I needed an answer to.

I think this is mainly because the AI is trained on the documentation and whenever the documentation is incomplete, it doesn’t have the knowledge to it. Downside it also that it will answer very confidently while basically guessing rather than saying “I don’t know the answer to this question, perhaps you can try if the following works…”.

One example I just tried is asking AI how I can get tomorrows or yesterdays date as a variable in workflows. For this you would need to get the current date and then add or subtract one day. There is no documentation for getting the current date (so the AI guesses something that does not work) and there is documentation on subtracting dates, where the documentation specifically mentions that workflows breaks if you try to generate a date that is before the current date, and the AI did not pick up this limitation and therefore suggests a solution that does not work.

When asking it to create a transform to concatenate two identity attributes it will give correct information, but then describing the issue takes more time than building it myself using the VS code plugin.

All in all I experience that when it comes to ISC development related questions, I spend more time talking to the AI than what I save in return.

Every now and then I will go back to the AI again to see if it has improved though, but currently I would recommend reading the documentation (training oneself on real SailPoint statements) rather than asking the AI questions (which taints your training by overly confident wrong answers).

Kind regards,
Angelo

7 Likes

Hey Jordan! I just submitted the idea as suggested, would love to see this in the future!

1 Like

@derek_putnam ! THIS! This is essentially what I was going to send to you, but Angelo has done a fantastic job of condensing the word vomit I had been working on. :sweat_smile: Thanks Angelo!

2 Likes

Just wanted to provided some feedback as well.
I don’t know all possible API’s that are available by heart, and the search in the developer portal is quite lacking (It’s quite hard to find the correct API), so since recently I try to ask which API to use or how to do a certain thing.

Oftentimes it goes well and it gives me some more info that I can get started with.
But I also had times where it completely invents things that it seems quite certain about.
Like this conversation:


It seems to produce a valid answer to my question, but as I was not finding this specific API I asked where to find the documentation and it linked me to a non-existing URL (which also seems like it might go somehere…)
https://developer.sailpoint.com/idn/api/v3/get-historical-exceptions

Either it’s ahead of it’s time and has access to unpublished, future API’s or there’s some hallucination going on… :slight_smile:

This is exactly right! We’re constantly working on improving our docs as a result of this, which of course is benefiting both our users directly as well as improving the AI. It is also trained only on docs on developer.sailpoint.com today, but we would like to extend that.

Yes, this is the unfortunate side effect of hallucinations of LLMs in AI, and I expect these foundational models will get better over time. We will also continue to improve our process based on your feedback here!

Remi, great to see you! Thanks so much for this feedback—incredibly detailed!

Could I reach out to you on this? I’d love to learn more? As you surmised, there is some hallucinating going on and something that we will continue to improve as ambassadors like you continue to help with!

This is awesome @derek_putnam !! :clap:t2:

1 Like