HIT L2: added data transformation into 10-minute chunks to L2

[leowerneck]

Change Summary

Closes #3327.

Overview

This PR moves the HIT macropixel 10-minute chunk transformation from L3 to L2. It coordinates with PR 161 in
imap_L3_processing that closes issue 143 of that repository.

The L2 macropixel product now consolidates the ten one-minute packetized records into a single record for each 10-minute
integration period. It also assigns the integration midpoint as the epoch and provides five-minute DELTA_MINUS and
DELTA_PLUS values through epoch_delta.

File changes

The following changes were made:

• imap_processing/hit/l2/hit_l2.py
  • Added transform_to_10_minute_chunks.
  • Consolidated each group of ten packetized records into one macropixel record containing all species and energy
    combinations.
  • Updated each consolidated record's epoch to the midpoint of its 10-minute integration period.
  • Added a five-minute epoch_delta for each record.
  • Updated process_macropixel_intensity to return the transformed dataset.
  • Configured the macropixel epoch attributes to use epoch_delta for both DELTA_MINUS_VAR and DELTA_PLUS_VAR.
• imap_processing/cdf/config/imap_constant_attrs.yaml
  • Added the CDF variable declaration for epoch_delta.

Testing

Ran the focused HIT L2 test suite:

  uv run pytest -q imap_processing/tests/hit/test_hit_l2.py

All 17 tests passed.

[tmplummer]

A few things to consider.

[tmplummer]

This looks good to me.

[jtniehof]

There are some general comments still to address scattered throughout the conversation.

Overall this looks good. Two things:

• Can the test of the time transformation be ported over from L3? There are no test updates in this PR, which makes me nervous
• Can you do an end-to-end test of making an L2 with this new code, making an L3 with that output, and comparing to the existing L3? That will I think be the acid test.

[leowerneck]

• Can the test of the time transformation be ported over from L3? There are no test updates in this PR, which makes me nervous

Fair point. I added imap_processing/tests/hit/test_hit_l2.py::test_transform_to_10_minute_chunks.

• Can you do an end-to-end test of making an L2 with this new code, making an L3 with that output, and comparing to the existing L3? That will I think be the acid test.

I used Claude (sorry for the verbosity) to create a convincing (at least to the untrained eye, like mine) end-to-end test in this repo.

Here's what you need to run (also found in the repo's README.md):

git clone https://github.com/leowerneck/L2_L3_acid_test.git
cd L2_L3_acid_test
./run_test.sh

If you also see "PASS", that would be reassuring. Let me know if there's anything missing from the test that you would like to see.

[jtniehof]

I'd feel more confident in something where we set up dependency .json and just ran imap_cli to make the files, so we're doing a full end-to-end on the actual files instead of transforming the L2 and just looking at the Python objects. I can take a stab later this afternoon.

[jtniehof]

imap-data-access download imap_hit_l2_macropixel-intensity-ed398e36-635a606f_20260819_v001.0003.json
#rename imap_hit_l2_macropixel-intensity-test_20260819_v001.0004.json, change minor version to 4
imap-data-access download imap_hit_l3_macropixel_20260819_v001.0005.cdf # for comparison
imap-data-access download imap_hit_l3_macropixel-6bf00ff0-1e59d123_20260819_v001.0005.json
# rename imap_hit_l3_macropixel-test_20260819_v001.0006.json, change minor version to 6, change minor version of input l2 to 4

imap_cli --instrument hit --data-level l2 --descriptor macropixel-intensity --start-date 20260819 --dependency imap_hit_l2_macropixel-intensity-test_20260819_v001.0004.json
python3 ./imap_l3_data_processor.py --instrument hit --data-level l3 --descriptor macropixel --start-date 20260819 --dependency imap_hit_l3_macropixel-test_20260819_v001.0006.json

cdfcompare -nonumber -noetc imap_hit_l3_macropixel_20260819_v001.0005.cdf imap_hit_l3_macropixel_20260819_v001.0006.cdf

Results are good:

Comparing "imap_hit_l3_macropixel_20260819_v001.0005.cdf" with "imap_hit_l3_macropixel_20260819_v001.0006.cdf"

    DIFFERENT value(s) for gEntry number 1 of attribute "Data_version"
    DIFFERENT number of elements for gEntry number 1 of attribute "File_naming_convention" (40 vs. 45)
    DIFFERENT value(s) for gEntry number 1 of attribute "Generation_date"
    DIFFERENT value(s) for gEntry number 1 of attribute "Logical_file_id"
    DIFFERENT value(s) for gEntry number 1 of attribute "Parents"
    DIFFERENT number of elements for gEntry number 1 of attribute "ground_software_version" (42 vs. 41)
    DIFFERENCES in 15 records for zVariable "epoch" (15 values total)

In these 15 records, the old epoch was 1 microsecond later (greater) than the new epoch. So there's probably some rounding going on involving use of datetimes. I generally prefer to do the math straight in TT2000 for that reason, but I don't think it's important in this case. However it might impact IMAP-Science-Operations-Center/imap_L3_processing/issues/121

The difference in File_naming_convention is apparently because the Great Renaming did not change that attribute on the existing files.

[leowerneck]

I created this repo which contains a helper to run the test you suggested in an automated way. This was a great exercise to understand how the pipeline runs, so thanks for the example!

Is updating the L2 file required by the Hit integration test (see imap_L3_processing::#161) something I can do, or is something I need to ask someone else to do?