vROps Webhook Payload for PagerDuty Alert Events Example

PagerDuty + VMware vRealize Operations Integration Benefits

  • Notify on-call responders based on alerts sent from vRealize Operations.
  • Send enriched event data from vRealize Operations including symptoms, properties/metrics of the impacted object and related objects and recommendations for actionable resolution.
  • Create high and low urgency incidents based on the severity of the event from the vRealize Operations event payload.
  • Provides updates based on changes to alert severity, state and ownership.
  • Incidents will automatically resolve in PagerDuty when the alert in vRealize Operations is automatically cancelled.

How it Works

  • vRealize Operations alerts can be configured with notification rules to send a webhook request to PagerDuty's Events API as an alert event type and create a new incident on the desired PagerDuty service.
  • If the alert state, status or ownership changes, the PagerDuty alert will be updated with the new information.
  • If the alert is cancelled automatically or manually in vRealize Operations, the corresponding alert will be resolved in PagerDuty.

Requirements

  • PagerDuty integrations require an Admin base role for account authorization. If you do not have this role, please reach out to an Admin or Account Owner within your organization to configure the integration.
  • Webhooks are supported in vRealize Operations Cloud or vRealize Operations 8.4 and higher.
  • You must have permission to create Payload Templates, Outbound Plugins and Notifications in vRealize Operations.

Support

If you need help with this integration, please contact VMware Global Support via VMware Customer Connect.

Integration Walkthrough

From a Browser

  1. Navigate to https://code.vmware.com/samples?id=7607.
  2. Click Download to begin download of the sample Webhook Payload template and save it for use in the vRealize Operations section.

    In PagerDuty

Integrating With a PagerDuty Service

  1. From the Configuration menu, select Services.
  2. There are two ways to add an integration to a service:
    • If you are adding your integration to an existing service: Click the name of the service you want to add the integration to. Then, select the Integrations tab and click the New Integration button.
    • If you are creating a new service for your integration: Please read our documentation in section Configuring Services and Integrations and follow the steps outlined in the Create a New Service section, selecting API as the Integration Type in step 4. Continue with the In vRealize Operations section (below) once you have finished these steps.
  3. Enter an Integration Name in the format monitoring-tool-service-name (e.g. vRealize Operations-Alerts) and select Use our API directly from the Integration Type menu (the option for Events API v2 should be selected).
  4. Click the Add Integration button to save your new integration. You will be redirected to the Integrations tab for your service.
  5. An Integration Key will be generated on this screen. Keep this key saved in a safe place, as it will be used when you configure the webhooks in vRealize Operations in the next section.

In vRealize Operations

The steps below will configure an example payload template and alert notification rule. You can use this example to create your own customized alert notifications for PagerDuty.

  1. From the Alerts tab, click Payload Templates under Configuration.
  2. Click the kabobs (three horizontal dots) under Payload Templates and select Import.
  3. Click BROWSE and locate the Payload Template-PagerDuty-Sample you downloaded earlier.
  4. When you select the template file, the import will begin. Wait until the status of the import indicates successful import.
  5. Click DONE to dismiss the import popup.
  6. You should see a new Payload Template in the list named PagerDuty Alert. If you do not, attempt step 3 again.
  7. Click Notifications under Configuration in the Alerts tab.
  8. Click ADD in Notification Settings to create a new rule.
  9. Enter PagerDuty Notification for the name and click NEXT.

You will now set the criteria for the notification. The following steps will only send the notifications for alerts on virtual machines with a guest OS file system that is approaching capacity. This is a default alert in vRealize Operations so it should easily trigger an alert. If you have disabled this alert, you can chose another alert in your system to trigger on, or change the criteria as desired. See the vRealize Operation documentation for help.

  1. Under Object Scope for Criteria select the Object Type and then type virtual machine into the Select an Object Type field. Select the vCenter Adapter > Virtual Machine object type from the pull-down list.
  2. Under Alert Scope select Category > Alert Definition.
  3. In the Alert Definitions popup, type guest file system into the Quick Filter and hit Enter.
  4. Drag the alert definition One or more virtual machine guest file systems are running out of disk space to the left.
  5. Click OK.
  6. Under Notify On select Cancelled, Updated, and New for Status.
  7. Click NEXT.
  8. For the Outbound Method select the Webhook Notification Plugin option.
  9. Click the CREATE NEW INSTANCE link.
  10. For Instance Name enter PagerDuty.
  11. For the URL enter https://events.pagerduty.com/v2/enqueue
  12. Click SAVE.
  13. Click NEXT.
  14. For the Payload Template select the PagerDuty Alert option.
  15. In the PagerDuty Integration Key input, enter the Integration Key you created in the PagerDuty section.
  16. Click CREATE.

When a new alert triggers for the alert definition One or more virtual machine guest file systems are running out of disk space the notification rule will send a webhook payload to PagerDuty. If you like you can test this by cancelling an existing alert for that definition and waiting for the alert to trigger again (no more than 5 minutes).

How to Uninstall

You can disable or remove this integration. Disable will only prevent the notifications from being sent. This is useful if you want to keep this in your system for testing or as an example to clone for other payloads.

To disable

  1. Under Alerts tab, select Configuration > Notifications.
  2. Click the kabobs (three vertical dots) next to the PagerDuty Notification.
  3. Click Disable.

To remove

  1. Under Alerts tab, select Configuration > Notifications.
  2. Click the kabobs (three vertical dots) next to the PagerDuty Notification.
  3. Click Delete.
  4. Click Payload Templates.
  5. Click the kabobs (three vertical dots) next to the template PagerDuty Alert.
  6. Click Delete.
  7. Under Administration tab, select Management > Outbound Settings.
  8. Click the kabobs (three vertical dots) next to the PagerDuty instance.
  9. Click Delete.

Sign in to be able to add comments.

Comments 1


4263907192 2 months ago
Hi John , i have problem when i configure the outbound instance , it has a test button (also has user/password in blank) , if i push test it fails even after the step 24 is set up the test button still not working , there is a alternative way to test if the webhook is working?