Skip to content

amtool: command to test receivers#4807

Draft
SoloJacobs wants to merge 18 commits intoprometheus:mainfrom
SoloJacobs:pull3491
Draft

amtool: command to test receivers#4807
SoloJacobs wants to merge 18 commits intoprometheus:mainfrom
SoloJacobs:pull3491

Conversation

@SoloJacobs
Copy link
Contributor

@SoloJacobs SoloJacobs commented Dec 12, 2025

This PR is an update of #3491 that's updated to work against main, which in turn is an update of #3139 .
So far, I have held off on making some changes myself, except for what was necessary to get CI green again. I also have a small laundry list of things I would fix here, but I would like a second pair of eyes on this request first.

This PR is an extension of @OktarianTB's 9-month old PR #3139 that's updated to work against the latest code in main, to hopefully increase its likelihood of being merged.

Credit for this work goes to @OktarianTB - I believe I've marked the commit authorship correctly, please let me know if I made any mistakes here.

This PR aims to make progress towards #2845 by adding a new command to amtool to send test notifications to every receiver in an alertmanager config file. The command takes in an alertmanager config file and optionally an alert file to mock values for labels and annotations. This is the MVP for the command, potentially a few other options can be added later if deemed of use. This largely re-uses the equivalent implementation in grafana/grafana.

I have only tested this with Slack so far.

OktarianTB and others added 18 commits December 12, 2025 08:44
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Co-authored-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Oktarian TB <oktotb@gmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Alex Weaver <weaver.alex.d@gmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
* Update logging
* Add argument to notify.NewIntegration

Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
Signed-off-by: Solomon Jacobs <solomonjacobs@protonmail.com>
@SoloJacobs SoloJacobs changed the title (WIP) amtool: command to test receivers amtool: command to test receivers Dec 12, 2025
@SoloJacobs SoloJacobs marked this pull request as ready for review December 12, 2025 16:00
@TheMeier
Copy link
Contributor

I gave it a try, its a useful helper. I know you did not add additional code on purpose yet. But I think we should at least include all notifiers that are in the current main branch. I noticed at least teamsv2 and rocketchat missing.

@TheMeier
Copy link
Contributor

@SoloJacobs can you rebase that?

Copy link
Contributor

@siavashs siavashs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SoloJacobs please write an elaborative PR description. While it is possible to go through the nested list of PRs, it is better to have all the context here and only link to the historical ones.

I think we are also missing the documentation for this.

I added a few other comments in the code as well.

@@ -0,0 +1,309 @@
// Copyright 2022 Prometheus Team
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Copyright 2022 Prometheus Team
// Copyright The Prometheus Authors

Comment on lines +268 to +270
for i, c := range nc.WebhookConfigs {
add("webhook", i, c, func(l *slog.Logger) (notify.Notifier, error) { return webhook.New(c, tmpl, l) })
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing this per receiver seems excessive to me.
We should instead just have one call to a module to add all the configured receivers.
Otherwise this logic will be always missing one or more receivers in future.

@@ -0,0 +1,119 @@
// Copyright 2022 Prometheus Team
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Copyright 2022 Prometheus Team
// Copyright The Prometheus Authors

@@ -0,0 +1,58 @@
// Copyright 2022 Prometheus Team
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Copyright 2022 Prometheus Team
// Copyright The Prometheus Authors

@SoloJacobs SoloJacobs marked this pull request as draft February 11, 2026 13:18
@siavashs siavashs added the keepalive Exempt from stale checks label Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

keepalive Exempt from stale checks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants