Idex release two updates by aldo9253 · Pull Request #3399 · IMAP-Science-Operations-Center/imap_processing

aldo9253 · 2026-08-19T17:58:33Z

Change Summary

Overview

Adds IDEX event classification and waveform saturation flags through L1A–L2A, with
saturation- and Science-mode-aware velocity and mass estimates.

File changes

Adds event and waveform saturation flag logic.
Propagates flags through L1A, L1B, and L2A.
Updates L2A velocity/mass calculations and masking behavior.
Adds CDF metadata and tests.

Testing

Relevant IDEX tests, formatting, linting, type checks, and pre-commit checks pass.
The local 20260719 processing chain completed successfully through L2A.

…dicate instrument status (science, pulser, noise capture). Added flag to indicate if the event was an actual dust event. Resolved ion grid velocity and mass estimates to use the QI/Qt ratio and Qt respectively. Added logic to NaN fields based on flag states.

Copilot

Pull request overview

Adds IDEX release-two event classification and saturation-aware processing across L1A–L2B.

Changes:

Adds event and waveform saturation flags through L1A–L2A.
Updates velocity, mass, and L2B count selection logic.
Adds CDF metadata, tests, and a local processing script.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`run_local_idex_chain.sh`	Adds local L1A–L2A processing workflow.
`imap_processing/idex/idex_event_flags.py`	Implements event and saturation classification.
`imap_processing/idex/idex_l1a.py`	Generates event flags.
`imap_processing/idex/idex_l1b.py`	Propagates flags to L1B.
`imap_processing/idex/idex_l2a.py`	Adds saturation-aware estimates and masking.
`imap_processing/idex/idex_l2b.py`	Filters events and selects target estimates.
`imap_processing/tests/idex/test_idex_event_flags.py`	Tests flag classification.
`imap_processing/tests/idex/test_idex_l0.py`	Updates expected variable count.
`imap_processing/tests/idex/test_idex_l1b.py`	Tests L1B flag propagation.
`imap_processing/tests/idex/test_idex_l2a.py`	Tests masking and Ion Grid estimates.
`imap_processing/tests/idex/test_idex_l2b.py`	Tests filtering and target selection.
`imap_processing/cdf/config/imap_idex_l1a_variable_attrs.yaml`	Defines L1A flag metadata.
`imap_processing/cdf/config/imap_idex_l1b_variable_attrs.yaml`	Defines L1B flag metadata.
`imap_processing/cdf/config/imap_idex_l2a_variable_attrs.yaml`	Defines L2A flag metadata.
`imap_processing/cdf/config/imap_idex_l2b_variable_attrs.yaml`	Corrects charge units to pC.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

lacoak21 · 2026-08-24T15:07:56Z

  VAR_TYPE: data

 # <=== Instrument Setting Attributes ===>
+science_event_flag:


Have you validated that these new attrs are ISTP compliant?

Alex is going to use SKTeditor to check

Best I can tell, these values are compliant. The webpage checker doesn't seem to like a fill-value of 255 but it's what we use across other variables so I think we leave it as is.

What was the error? Can you attach a screen shot? 255 may be a valid fill val for other variables but not these if its not configured properly.

ISTP checker now returning compliance for the flags! Thanks for catching those issues.

excellent. Super appreciate you checking.

lacoak21

OK I have reviewed part of this and will continue when comments are addressed. In the future this PR could have been broken down into maybe 4 or 5 PRs. We generally try to make PRs as small as possible.

lacoak21 · 2026-08-25T15:15:56Z

+        saturation_flag = f"{waveform_name}_saturation_flag"
+        if saturation_flag not in l2a_dataset:
+            message = f"Required L2A saturation flag is missing: {saturation_flag}"
+            logger.error(message)


I think you can just choose one. Either log a message and continue with the code and do something OR raise an error.

lacoak21

LGTM! Once we know these are ISTP compliant I think this is good to go.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

lacoak21 · 2026-08-25T16:35:58Z

+  CATDESC: Boolean event classification flag.
+  DEPEND_0: epoch
+  DICT_KEY: SPASE>Support>SupportQuantity:QualityFlag
+  FILLVAL: 0


Ah this is super important @aldo9253 I would definitely fix this

+    right_bracketed = (
+        right < corrected.size - 1
+        and np.isfinite(corrected[right - 1])
+        and corrected[right - 1] >= half_height
+        and np.isfinite(corrected[right])
+        and corrected[right] < half_height
+    )


aldo9253 · 2026-08-25T19:29:21Z

[cid:2fbfdab1-f6a5-4470-85da-a8b21965989e]

________________________________ From: Luisa Coakley ***@***.***> Sent: Tuesday, August 25, 2026 1:27 PM To: IMAP-Science-Operations-Center/imap_processing ***@***.***> Cc: aldo9253 ***@***.***>; Mention ***@***.***> Subject: Re: [IMAP-Science-Operations-Center/imap_processing] Idex release two updates (PR #3399) [External email - use caution] @lacoak21 commented on this pull request.

________________________________ In imap_processing/cdf/config/imap_idex_l1b_variable_attrs.yaml<#3399 (comment)>:

@@ -56,6 +64,66 @@ spice_base: &spice_base

VAR_TYPE: data # <=== Instrument Setting Attributes ===> +science_event_flag: What was the error? Can you attach a screen shot? 255 may be a valid fill val for other variables but not these if its not configured properly. — Reply to this email directly, view it on GitHub<#3399?email_source=notifications&email_token=AJRWSSWVHFP3AMEJNZCU6L35LXSC3A5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSGMYTANBXGQ32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#discussion_r3856512975>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJRWSSQUEAECSY6LAYOBZL35LXSC3AVCNFSNUABFKJSXA33TNF2G64TZHM3DKNBWG44TQMJYHNEXG43VMU5TKMJZGU3DGOJXGY4KC5QC>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AJRWSSU2B3GVY5AOE5MFBYT5LXSC3A5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSGMYTANBXGQ32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/AJRWSSSMQ6ZWF4YCYOTI3DL5LXSC3A5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKMBSGMYTANBXGQ32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

lacoak21 · 2026-08-25T19:33:41Z

[cid:2fbfdab1-f6a5-4470-85da-a8b21965989e]

Can you try uploading it again?

aldo9253 · 2026-08-25T19:39:45Z

Just emailed it to you.

lacoak21 · 2026-08-25T19:46:28Z

  VALIDMIN: 0
  VAR_TYPE: support_data

+event_flag_base: &event_flag_base


Oh I see. There are a couple of issues here.

I would not use "trigger_base" as the base because those are for variables that are not quality flags. So I would remove that part.

Second, make sure the array is actually np.unit8 e.g. data.astype(np.uint8)
It thinks the data is np.uint64 as implied by the warning.

aldo9253 · 2026-08-25T20:06:10Z

Okay, hopefully this commit addressed it all.

lacoak21 · 2026-08-25T20:28:51Z

Okay, hopefully this commit addressed it all.

Nice looks great.

Another thing I thought of. Have you produced a CDF with this code locally?

aldo9253 · 2026-08-25T20:33:55Z

Yes, for every version. Here are the most recent versions!

…

________________________________ From: Luisa Coakley ***@***.***> Sent: Tuesday, August 25, 2026 2:29 PM To: IMAP-Science-Operations-Center/imap_processing ***@***.***> Cc: aldo9253 ***@***.***>; Mention ***@***.***> Subject: Re: [IMAP-Science-Operations-Center/imap_processing] Idex release two updates (PR #3399) [External email - use caution] [https://avatars.githubusercontent.com/u/48064300?s=20&v=4]lacoak21 left a comment (IMAP-Science-Operations-Center/imap_processing#3399)<#3399 (comment)> Okay, hopefully this commit addressed it all. Nice looks great. Another thing I thought of. Have you produced a CDF with this code locally? — Reply to this email directly, view it on GitHub<#3399?email_source=notifications&email_token=AJRWSSREBGU7POQ5KN2F3FL5LXZJTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRGYZTIMRWHAZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5416342682>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJRWSSUTXQZ5DHRY3QFHHVT5LXZJTAVCNFSNUABFKJSXA33TNF2G64TZHM3DKNBWG44TQMJYHNEXG43VMU5TKMJZGU3DGOJXGY4KC5QC>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AJRWSSXQICM3RU24URSB5QL5LXZJTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRGYZTIMRWHAZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/AJRWSSXPVCPEESG42AVHWJ35LXZJTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBRGYZTIMRWHAZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

aldo9253 · 2026-08-25T21:38:52Z

I have some l2b changes here that will conflict with your 10-day refactor. I'm going to revert the ones here so that we have a clean merge. We will tackle all l2b related updates in seperate PRs more cleanly.

aldo9253 added 6 commits August 18, 2026 10:47

Add IDEX event classification flags

2f10bf8

Merge remote-tracking branch 'origin/dev' into idex_release_two_updates

e761beb

fixed expected nan condition in l2a for ion grid values

204e3c9

added event category masking to l2b/c, left NaN block in place.

76a4275

fixed description in idex_event_flags.py

fec1ed1

bryan-harter requested review from lacoak21 and a balanced review from Copilot August 20, 2026 19:49

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Comment thread imap_processing/idex/idex_l2b.py Outdated

Comment thread imap_processing/idex/idex_event_flags.py Outdated

Comment thread imap_processing/idex/idex_event_flags.py Outdated

Comment thread run_local_idex_chain.sh Outdated

Comment thread run_local_idex_chain.sh Outdated

Copilot started reviewing on behalf of bryan-harter August 20, 2026 23:03 View session

This was referenced Aug 21, 2026

IDEX dust identification algorithm #3336

Closed

IDEX L1B TOF unit correction #3404

Closed

lacoak21 assigned aldo9253 Aug 24, 2026

lacoak21 reviewed Aug 24, 2026

View reviewed changes

Comment thread imap_processing/idex/idex_l2a.py Outdated

Comment thread imap_processing/idex/idex_l2a.py

Comment thread imap_processing/idex/idex_l2a.py Outdated

Comment thread run_local_idex_chain.sh Outdated

aldo9253 added 5 commits August 24, 2026 11:17

addressed round of comments

118d907

addressed round of comments

69b9c08

added log and mandatory state for saturation flags

1c80d14

re-run checks

8566c76

fixed tests

0f16d72

lacoak21 reviewed Aug 25, 2026

View reviewed changes

Comment thread imap_processing/tests/idex/test_idex_l0.py

lacoak21 reviewed Aug 25, 2026

View reviewed changes

lacoak21 approved these changes Aug 25, 2026

View reviewed changes

lacoak21 requested a balanced review from Copilot August 25, 2026 16:23

Copilot started reviewing on behalf of lacoak21 August 25, 2026 16:24 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

corrected fill values

74db977

lacoak21 reviewed Aug 25, 2026

View reviewed changes

addressed data type issue

c433242

aldo9253 added 4 commits August 25, 2026 15:48

reverted l2b changes for cleaner merging

6674c4c

Merge upstream dev into idex_release_two_updates

06cc24d

Use upstream mA conversions for IDEX TOF channels

eeff14b

pre-commit fix

0165f7f

tmplummer merged commit f469be9 into IMAP-Science-Operations-Center:dev Aug 28, 2026
38 of 39 checks passed

Uh oh!

Conversation

aldo9253 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Overview

File changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lacoak21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lacoak21 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aldo9253 commented Aug 25, 2026 via email

Uh oh!

lacoak21 commented Aug 25, 2026

Uh oh!

aldo9253 commented Aug 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aldo9253 commented Aug 25, 2026

Uh oh!

lacoak21 commented Aug 25, 2026

Uh oh!

aldo9253 commented Aug 25, 2026 via email

Uh oh!

aldo9253 commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aldo9253 commented Aug 19, 2026 •

edited

Loading