Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| def _getIdHash(id): | ||
| """Return md5 prefix based on id value""" | ||
| m = hashlib.new("md5") | ||
| m.update(id.encode("utf8")) |
Check failure
Code scanning / CodeQL
Use of a broken or weak cryptographic hashing algorithm on sensitive data
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI over 1 year ago
To fix the issue, replace the use of the MD5 hashing algorithm in _getIdHash with a stronger algorithm, such as SHA-256. This ensures that the hash function is resistant to pre-image and collision attacks. The change involves:
- Updating the
_getIdHashfunction to usehashlib.sha256instead ofhashlib.new("md5"). - Ensuring that the rest of the code remains functional by keeping the truncation to the first 5 characters of the hash.
No additional imports are required since hashlib already supports SHA-256.
| @@ -25,4 +25,4 @@ | ||
| def _getIdHash(id): | ||
| """Return md5 prefix based on id value""" | ||
| m = hashlib.new("md5") | ||
| """Return sha256 prefix based on id value""" | ||
| m = hashlib.sha256() | ||
| m.update(id.encode("utf8")) |
mattjala
left a comment
mattjala
left a comment
There was a problem hiding this comment.
Besides a few minor comments and questions, this is good to go in. I'll try to get the outstanding PRs on hdf5-json reviewed this week so that we can avoid having HSDS depend on a specific branch.
| @@ -0,0 +1,2973 @@ | |||
| openapi: 3.1.0 | |||
There was a problem hiding this comment.
The README points at HDFGroup/hdf5-rest-api as an authoritative description of the API, but it's now out of date. We should remove any references to it and flag it as out of date.
| # link related functions | ||
| # | ||
| from h5json.time_util import getNow | ||
| from h5json.link_util import validateLinkName, getLinkClass, getLinkPath, getLinkFilePath |
There was a problem hiding this comment.
validateLinkName and isEqualLink moved to h5json.link_util, but their analogues for attributes (validateAttributeName and isEqualAttr) stayed behind in hsds/util/attrUtil.py. Is there a reason the two are split differently?
| multiple groups in one request. `type` is not permitted in the body | ||
| (groups have no datatype). | ||
|
|
||
| **Bug:** `implicit` is only forwarded to argument construction for |
There was a problem hiding this comment.
Should be moved to a GH issue instead of a note in openapi
| (multi-item list) create; it is not restricted to non-batch | ||
| creates. | ||
|
|
||
| **Bug:** for a batch create (list with more than one item), |
There was a problem hiding this comment.
Should be moved to a GH issue instead of being in the openapi spec
| schema: { type: boolean, default: false } | ||
| description: | | ||
| Include an `alias` list of h5paths that resolve to this | ||
| dataset. Bug: on this specific route, the flag that gates |
There was a problem hiding this comment.
Bug desc should be moved to a GH issue
| properties: | ||
| bytes_sent: | ||
| type: integer | ||
| description: "Note: a source-level bug (hsds/basenode.py) assigns bytes_recv over this key immediately after setting it, so this actually reports received bytes, and sent-byte count is not exposed." |
There was a problem hiding this comment.
Bug desc should be moved to a GH issue
| description: | | ||
| Include an `alias` list of h5paths that resolve to this group. | ||
| Note: parsed with a raw truthiness check rather than real | ||
| boolean parsing, so `?getalias=0` is truthy and turns this on. |
There was a problem hiding this comment.
Truthiness issue seems like a bug and should be moved from here to a GH issue
| description: | | ||
| Include an `alias` list of h5paths that resolve to this | ||
| datatype. Note: parsed with a raw truthiness check, so | ||
| `?getalias=0` is truthy and turns this on. |
There was a problem hiding this comment.
Truthiness issue seems like a bug and should be moved from here to a GH issue
6 participants
Use h5json package for typing and objids
Important
Migrated HSDS to use the h5json library for core utilities, restructured utility modules, added support for client-provided object IDs and timestamps, and updated dependencies to require Python 3.10+ with h5json 1.0.0+.
Library Migration and Utility Restructuring
hsds/util/idUtil.py,hsds/util/timeUtil.py,hsds/util/hdf5dtype.py, andhsds/util/arrayUtil.pyas their functionality is now provided by h5json.hsds/util/nodeUtil.pywith node ID generation, partitioning, and datanode URL resolution functions.util.idUtil�h5json.objid,util.timeUtil�h5json.time_util).Object ID and Timestamp Handling
POST_Dataset,POST_Group,POST_Datatype, and related functions indset_dn.py,group_dn.py,ctype_dn.py, anddset_sn.py.max_timestamp_driftconfiguration parameter to validate client-provided timestamps inattr_dn.py,link_dn.py, and related modules, with fallback to server-generated timestamps when skew exceeds threshold.deleted_idsset when creating new objects with the same ID.Configuration and Dependencies
default_vlen_type_size,predate_maxtime,posix_delay,max_compact_dset_size, andmax_timestamp_drifttoadmin/config/config.yml.pyproject.tomlto require Python 3.10+, add h5json 1.0.0+, update numpy to 2.0.0+, and constrain numcodecs to �0.15.1..github/workflows/python-package.yml.API and Function Refactoring
POST_Dataset,POST_Group, andPOST_Datatypehandlers to support batch creation of multiple objects using new helper functions (createDatasets,createGroups,createDatatypeObjs) andDomainCrawlerfor writing initial data.getChunkLayoutcalls togetChunkDimsthroughout codebase; moved layout from top-level response to nested undercreationProperties.h5domaintofileinlink_dn.py,link_sn.py, andservicenode_lib.py; added per-link timestamp validation inPUT_Links.POST_Dataset,POST_Group, andPOST_Datatypeinstead of always creating empty objects.New Functionality
hsds/post_crawl.pywithPostCrawlerclass for asynchronously creating multiple HDF5 objects with configurable worker count and error handling.getConsolidatedMetaDatafunction inasync_lib.pyto create consolidated metadata summaries for all objects in a domain.put_datamethod toDomainCrawlerfor writing one-chunk dataset values; addeddoPointWriteanddoHyperslabWritefunctions indset_lib.pyfor writing point and hyperslab selections.getobjsparameter togetDomainResponsefunction to optionally return domain objects from S3 summary file.Bug Fixes and Improvements
HTTPInternalServerErrortoHTTPBadRequestfor duplicate object IDs and invalid configurations inctype_dn.py,dset_dn.py, andgroup_dn.py.posix_delayconfiguration support tofileClient.pyfor simulating cloud storage latencies inget_object,put_object, andlist_keysmethods.HSDS_VERSIONfrom 0.9.2 to 1.0.0 inbasenode.py.Test Updates
testPostDatasetWithId,testPostTypeWithId,testPostWithId), attribute initialization (testPostDatasetWithAttributes,testPostWithAttributes), timestamp handling (testUseTimestamp), and batch creation (testPostMulti,testDatasetPostMulti).creationPropertiesinstead of top-level; removedCHUNK_MIN/CHUNK_MAXconstants and moved them to local scope; updated external link tests to usefilefield instead ofh5domain.array_util_test.py,hdf5_dtype_test.py, andid_util_test.pyas their functionality is now tested through h5json library.createObjId,getFilterItem) instead of local utilities.This description was created by
for 2bafb51. You can customize this summary. It will automatically update as commits are pushed.