25 tarin on three year data

[rogerkuou]

fix #25

relates #88
closes #89

🔴 the branch #84 has been merged into this branch, wait for it...

did not finish the train-validation-test split in this PR, but made a new issue #28

[rogerkuou]

Hi @SarahAlidoost and @meiertgrootes, I created an exmaple training process on a subset of the two year data., and ran it on Levante. In this PR I included example SLURM training process, with an README on how to config the jobs on Levante.

A copy of the example run can be found on /work/bd0854/b380854/eso4clima. I executed the slurm task on my home dir, and copied the entire experiment here.

[SarahAlidoost]

@rogerkuou thanks for the script. Since the PR #29 fixed a few issues, we need to merge main into this branch. I also left some comments, mainly about the structure of the example.py and the code that should be run with slurm. If something is unclear, please let me know. In meantime, I will work on issue #33.

[rogerkuou]

Hi @SarahAlidoost, thanks for the review! I implemented most of your comments:

1. I separated the example script to two: training and inference
2. Now the training scipt export models with checkpoint. I slightlt modified the class to make it returning the config
3. I used logging to replace the print statements
4. The plotting part has been removed

I did not implemente the training utility function and will leave it to #33 .

Can you give another look?

[SarahAlidoost]

@rogerkuou thanks for addressing the comments 👍 . Here some more suggestions:

• I see that the example notebook has been changed in this PR. I cannot see exactly what is changed, but since this PR is about testing large data on HPC, let's not change the example notebook.
• No need to add inference script in this PR. For now we can skip that one. Let's focus on setup of the training on HPC in this PR. Also, in fixing #32 we can add inefrence script later.
• After implementing these suggestions and re-running the slurm job, can you please add the slurm logfile to the PR as well? Also, can you perhaps give an indication how much resources have been used to complete the job.

If something not clear, please let me know.

[SarahAlidoost]

Instead of hardcoding, can we get these parameters directly from the results of the hyper parameter tuning?