Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle junk bytes after tree #1822

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

zacharyburnett
Copy link
Member

@zacharyburnett zacharyburnett commented Aug 12, 2024

Description

fixes asdf-format/asdf-standard#441

  • junk before blocks (after tree) raises OSError: Invalid block magic

    asdf/asdf/_block/reader.py

    Lines 249 to 257 in 867c280

    try:
    # load first and last blocks to check if the index looks correct
    for index in (0, -1):
    fd.seek(block_index[index])
    buff = fd.read(magic_len)
    if buff != constants.BLOCK_MAGIC:
    msg = "Invalid block magic"
    raise OSError(msg)
    blocks[index].load()
  • junk after blocks (after tree) raises asdf.exceptions.AsdfBlockIndexWarning: Invalid block index contents for block 0, falling back to serial reading: Invalid block magic

    asdf/asdf/_block/reader.py

    Lines 258 to 262 in 867c280

    except (OSError, ValueError) as e:
    msg = f"Invalid block index contents for block {index}, falling back to serial reading: {e!s}"
    warnings.warn(msg, AsdfBlockIndexWarning)
    fd.seek(starting_offset)
    return _read_blocks_serially(fd, memmap, lazy_load, after_magic)

Checklist:

  • pre-commit checks ran successfully
  • tests ran successfully
  • for a public change, a changelog entry was added
  • for a public change, documentation was updated
  • for any new features, unit tests were added

@zacharyburnett zacharyburnett self-assigned this Aug 12, 2024
@zacharyburnett zacharyburnett changed the title add tests for junk bytes handle junk bytes after tree Aug 12, 2024
@zacharyburnett
Copy link
Member Author

zacharyburnett commented Aug 15, 2024

we've decided to move these reference files to asdf-standard, and cover the following cases:

  • 0 bytes
  • 5 bytes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

A file with junk bytes after the tree leads to a read failure
1 participant