Hello - we noticed a workflow in production fail this morning with the below “panic” error.
Has anyone seen this error before?
WorkflowExecutionFailed 2024-03-06T15:06:27.995744411Z {"error":"Workflow failed on panic due to FailWorkflow workflow panic policy"}
Interesting, I haven’t seen such error. I am assuming this might be because of Today’s Service Degradation.
Is your tenant in US-East-1?
Yes, we are in that region. Thanks!
It must be that problem. Here’s what I find online (Go Dev) :
WorkflowPanicPolicy is used for configuring how worker deals with workflow code panicking. The default behavior is to block workflow execution until the problem is fixed.
Maybe due to degradation the policy stopped all the workers which blocked the execution
That’s a good guess - but the status page is showing all clear as of 8:45 UTC, we saw this error at 15:06 UTC
Monitoring - All service are currently operational and restored.
We are currently monitoring.
Mar 06, 2024 - 08:45 UTC
I don’t think the issue got resolved at 8:45 UTC because I was not able to login to my tenant until 15:45 - 16 UTC
Thanks @colin_mckibben for reaching out.
No - we only saw it the one time.
It seemed like a really odd error message. Any insight into what might have happened here?
