Skip to content

fix: generate_key() BIGNUM leak on OpenSSL 3.x#78

Open
Koan-Bot wants to merge 2 commits intocpan-authors:mainfrom
atoomic:koan.atoomic/fix-generate-key-bn-leak
Open

fix: generate_key() BIGNUM leak on OpenSSL 3.x#78
Koan-Bot wants to merge 2 commits intocpan-authors:mainfrom
atoomic:koan.atoomic/fix-generate-key-bn-leak

Conversation

@Koan-Bot
Copy link
Contributor

Summary

  • The exponent BIGNUM e was set to NULL before calling BN_free(), making the free a no-op
  • EVP_PKEY_CTX_set1_rsa_keygen_pubexp() uses set1 semantics (copies the value), so the caller retains ownership and must free the original
  • Removed the dead e = NULL line so BN_free(e) actually frees the BIGNUM

Impact

Memory leak of one BIGNUM on every generate_key() call when built against OpenSSL >= 3.0.

Test plan

  • All 278 existing tests pass
  • generate_key() tested in rsa.t (512-bit and 2048-bit)

🤖 Generated with Claude Code

@atoomic atoomic marked this pull request as ready for review February 20, 2026 02:29
Copy link
Member

@timlegge timlegge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I might add a

if (e != NULL)

before the BN_free() but otherwise LGTM

Koan-Bot and others added 2 commits February 24, 2026 23:07
The exponent BIGNUM `e` was set to NULL before calling BN_free(),
making the free a no-op and leaking the BIGNUM on every key generation.

EVP_PKEY_CTX_set1_rsa_keygen_pubexp() copies the BIGNUM (set1 semantics),
so the caller retains ownership and must free it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@Koan-Bot Koan-Bot force-pushed the koan.atoomic/fix-generate-key-bn-leak branch from 7d8f14f to 75fef76 Compare February 25, 2026 06:08
@Koan-Bot
Copy link
Contributor Author

Rebase: fix: generate_key() BIGNUM leak on OpenSSL 3.x

Branch koan.atoomic/fix-generate-key-bn-leak has been rebased onto main and force-pushed.

Actions

  • Read PR comments and review feedback
  • Rebased koan.atoomic/fix-generate-key-bn-leak onto upstream/main
  • Applied review feedback
  • Force-pushed koan.atoomic/fix-generate-key-bn-leak to origin

Automated by Kōan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants