Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Annealing configs #738

Merged
merged 97 commits into from
Nov 1, 2024
Merged

Annealing configs #738

merged 97 commits into from
Nov 1, 2024

Conversation

dirkgr
Copy link
Member

@dirkgr dirkgr commented Oct 21, 2024

This is mainly a bunch of annealing configs that right now live in an obscure branch.
There are also run scripts to make these work in various scenarios.

epwalsh and others added 30 commits July 25, 2024 11:46
@epwalsh
32 nodes
0f586f4
@epwalsh
add back activation checkpointing
844cadd
@epwalsh
try fine-grained act checkpointing
020d3af
@epwalsh
add load path
64e1a57
@epwalsh
Use local copy of tokenizer
e5d4407
@epwalsh
try only checkpointing the embeddings
efc8068
@epwalsh
Revert "try only checkpointing the embeddings"
c47736e 
This reverts commit efc8068.
@epwalsh
try one-in-four
f40f94d
@epwalsh
try one-in-eight
4107347
@epwalsh
run on 64 nodes
390b290
@epwalsh
hybrid sharding
5563fd4
@epwalsh
back to 32 for now
888753e
@epwalsh
try one-in-four again
1531c60
@epwalsh
setup for 64 GPUs
ef33ff1
@epwalsh
back to 32
080e04c
@epwalsh
okay, 16 nodes
f53f55a
@epwalsh
back to 64 nodes
453e34d
@dirkgr
Peteish S3
4d2e82a
@dirkgr
Script to run Peteish7 on LUMI
ac53fa0
@dirkgr
LUMI has modernized
1992b98
@dirkgr
Better path management
6ccd4ae
@dirkgr
Save checkpoints remotely
3df55f0
@dirkgr
Don't need this
97dc472
@dirkgr
Fused loss is not available on LUMI
67fd4e8
@dirkgr
Fix container used for logging into nodes
68a2a3a
@dirkgr
Try Shane's setup for real
446d064
@dirkgr
Fix library paths
854616e 
Peak computer science right here
@dirkgr
Add in settings that work
da2ac01
@dirkgr
Adds a peteish13 config
8d99d42
@dirkgr
LUMI script for peteish13
fd7e339
soldni and others added 22 commits September 4, 2024 22:55
@soldni
added olmoe data run
2723531
@soldni
renamed
2d748fd
@soldni
fixing paths
8739adc
@soldni
sigh do no trust autocomplete with paths
0b8f419
@dirkgr
Adds a constant-with-warmup scheduler
b9d61a6
@dirkgr
Merge branch 'dave/annealing_peteish_v2' of https://github.com/allena…
851e284 
…i/LLM into dave/annealing_peteish_v2
@dirkgr
Adds a config to do WSD annealing
f2a9e56
@dirkgr
Rename, because we don't need a token count anymore
54f154b
@dirkgr
Add a really short anneal
090b393
@dirkgr
Shard better
6d395d7
@dirkgr
A config for continuing an interrupted run
b9c6775
@dirkgr
Revert "Shard better"
048c809 
This reverts commit 6d395d7.
@dirkgr
Config to D the 7B WSD anneal
d028d88
@dirkgr
Merge remote-tracking branch 'origin/main' into annealing_peteish_v2_…
b83ce02 
…neweval

# Conflicts:
#	olmo/config.py
#	olmo/optim.py
@dirkgr
New evals
92139f2
@dirkgr
Less aggressive LR for the WSD anneal
149b550
@dirkgr
D for the medium LR WSD config
8cccfa0
@dirkgr
Config to continue one of the WSD anneals
838423e
@dirkgr
Merge branch 'main' into annealing_peteish_v2_neweval
955a02b
@dirkgr
Productivity through formatting
a49b5b3
@dirkgr
Changelog
52984d7
@dirkgr
Merge branch 'main' into annealing_peteish_v2_neweval
7e34173
epwalsh
epwalsh approved these changes Nov 1, 2024
View reviewed changes
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a lot of configs!

2015aroras
2015aroras approved these changes Nov 1, 2024
View reviewed changes
Copy link
Collaborator

@2015aroras 2015aroras left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a lot

@dirkgr
Copy link
Member Author

dirkgr commented Nov 1, 2024

It's accumulated over quite a long time. I think we may want to cull these at some point.

@dirkgr dirkgr merged commit afd728f into main Nov 1, 2024
12 checks passed
@dirkgr dirkgr deleted the annealing_peteish_v2_neweval branch November 1, 2024 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@2015aroras 2015aroras 2015aroras approved these changes

@epwalsh epwalsh epwalsh approved these changes

@AkshitaB AkshitaB Awaiting requested review from AkshitaB

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dirkgr @epwalsh @2015aroras @dwadden @soldni