Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error on unpickling TextLogger after deleting directory where text log was saved. #1661

Open
man2machine opened this issue Aug 2, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@man2machine
Copy link

man2machine commented Aug 2, 2024

🐛 Describe the bug
Summary: When reloading a strategy checkpoint using maybe_load_checkpoint, the loggers for that strategy are unpickled, which calls TextLogger._fobj_deserialize. If the strategy saved text logs to a directory that no longer exists (for example in a temporary directory), then when loading the checkpoint you get FileNotFoundError for any file definition of the form "path:<path to file>".

🐜 To Reproduce

os.makedirs("logs", exist_ok=True)
log_fname = "logs/{}.log".format(datetime.now().strftime('%Y-%m-%d_%H-%M-%S'))
 
text_logger = TextLogger(file=open(log_fname, 'w'))
eval_plugin = EvaluationPlugin(
    loggers=[text_logger]
)

strategy = Naive(
    model=model,
    optimizer=optimizer,
    criterion=criterion,
    evaluator=eval_plugin,
)

save_checkpoint(
    strategy,
    "checkpoint.pt"
)

shutil.rmtree("logs")

strategy, _ = maybe_load_checkpoint(
    strategy,
    "checkpoint.pt"
)

🐝 Expected behavior
The log directory and file should be created. This is special pickling behavior from avalanche, hence ideally it should either handle all the pickling cases properly or not pickle the text logger.

@man2machine man2machine added the bug Something isn't working label Aug 2, 2024
@AntonioCarta
Copy link
Collaborator

Right now loggers add metrics to existing files. The idea is that you want the file to have the complete history. If the file does not exist this is not possible. To me this is expected behavior. However, I'm open to an alternative that starts the log from scratch if the file is not available anymore, as long as it doesn't add too much complexity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants