feat: add Kafka extractor and DSM - #837
jeastham1993 wants to merge 4 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c9f4477ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Self-managed Kafka events arrive with eventSource "aws:SelfManagedKafka", so the previous comparison against "SelfManagedKafka" never matched. Those invocations fell through to generic Lambda-context extraction and lost both Kafka header propagation and DSM checkpointing. Also stop skipping the DSM consume checkpoint when a record has no headers. Guard only on the topic and pass the nullable carrier through, matching the SQS/Kinesis extractors so uninstrumented producers still get consume nodes and pathway roots.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a98cf8d47
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6ca569e8e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| static isKafkaEvent(event: any): event is MSKEvent | SelfManagedKafkaEvent { | ||
| return ( | ||
| (event.eventSource === "aws:kafka" || event.eventSource === "aws:SelfManagedKafka") && |
There was a problem hiding this comment.
For self-managed Kafka event source mappings, Lambda emits eventSource: "SelfManagedKafka" (without the aws: prefix), matching the SelfManagedKafkaEvent shape imported above. This predicate therefore rejects those events, so they fall through to generic Lambda-context extraction and receive neither Kafka header propagation nor DSM checkpoints; accept the unprefixed value (or both variants).
Useful? React with 👍 / 👎.
|
Hold off merging this PR until 830 is merged. PR 830 is a community contribution that adds the extraction support, but doesn't add DSM support. Merge the two together once the first is complete. |
2 participants
What does this PR do?
Add support for extraction and DSM checkpointing for Kafka. This supports the default Kafka event structure, but noted that if customers have some custom configuration this won't work - https://docs.aws.amazon.com/lambda/latest/dg/services-consume-kafka-events.html#services-consume-kafka-events-payload
Motivation
Adding support for all Lambda ESM's.
Testing Guidelines
Unit tests added to test functionality, tested with custom deployed version of layer.
Additional Notes
Hold off merging this PR until 830 is merged. PR 830 is a community contribution that adds the extraction support, but doesn't add DSM support. Merge the two together once the first is complete.
Types of Changes
Check all that apply