Map syncode version to transformers version

README.md

@@ -62,6 +62,14 @@ Simply install SynCode via PyPi using the following command:
 ``` bash
 pip install git+https://github.com/uiuc-focal-lab/syncode.git
 ```
+
+Note: SynCode depends on HuggingFace [transformers](https://github.com/huggingface/transformers):
+| SynCode version | Recommended transformers version |
+| -------------- | -------------------------------- |
+| `v0.1.4` (latest) | `v4.44.0`                         |
+| `v0.1.2`          | `v4.42.0`                         |
+
+
 ### Usage option 1:
 SynCode can be used as a simple logit processor with HuggingFace [transformers](https://github.com/huggingface/transformers) library interface. Check this [notebook](./notebooks/example_logits_processor.ipynb) for example.
 

requirements.txt

@@ -3,7 +3,7 @@ interegular
 regex==2023.8.8
 torch
 tqdm
-transformers
+transformers==4.44.0
 mxeval @ git+https://github.com/shubhamugare/mxeval.git
 datasets
 jsonschema

syncode/evaluation/json_eval.py

@@ -35,8 +35,12 @@ def run_json_eval(
 
         for task_id, problem in enumerate(problems):
             output = JSONEval.run_eval_for_task(syncode, syncode.num_samples, problem, samples, pbar, task_id, prompt_type=prompt_type)
+
             if debug_task_id is not None:
+                # This is for debugging purposes
+                logger.log_code(msg="Problem", code=output[0])
                 return output
+
             outputs.append(outputs) 
 
         schema_result = validate_json_data(syncode, samples, results, logger=logger)