Update Replication Instructions to Use Script to Load Pretrained Model

Replication.txt

@@ -22,7 +22,7 @@ In order to run the training process with our specific model, we need to make a
 
 To alter the sentence-transformers library, clone the repository from https://github.com/UKPLab/sentence-transformers locally and replace the SentenceTransformer.py and Transformer.py files located within the sentence-transformers/sentence_transformers/ and sentence-transformers/sentence_transformers/models/ directories of the cloned repository, respectively, with those located inside dev/ folder. (This has already been done in this notebook instance, but this will have to be completed if training on another system.)
 
-Before conducting actual training, we also need to clone the mosaic-bert-base-seqlen-2048 model locally and make a few small changes to its config.json file. Running Mosaic_Model.ipynb will execute this process and get our model ready to begin training. (Again, this has already been done in this notebook instance, but this will have to be completed if training on another system.)
+Before conducting actual training, we also need to clone the mosaic-bert-base-seqlen-2048 model locally and make a few small changes to its config.json file. Running load_mosiac.py will execute this process and get the model ready to begin training. (Again, this has already been done in this notebook instance, but this will have to be completed if training on another system.)
 
 Training
 

load_mosaic.py

@@ -0,0 +1,23 @@
+from huggingface_hub import snapshot_download
+import json
+import os
+
+REPO_ID = "mosaicml/mosaic-bert-base-seqlen-2048"
+MODEL_DIRECTORY = "mosaic-bert-base-seqlen-2048"
+
+
+def main():
+    snapshot_download(repo_id=REPO_ID, local_dir=MODEL_DIRECTORY)
+
+    # modify the model's config.json file to satisfy our requirements
+    config_file_path = os.path.join(MODEL_DIRECTORY, 'config.json')
+    contents = json.load(open(config_file_path))
+    contents['architectures'] = ['BertModel']
+    contents['auto_map']['AutoModel'] = 'bert_layers.BertModel'
+    contents['torch_dtype'] = 'bfloat16'
+    contents['transformers_version'] = '4.28.1'
+    contents['_name_or_path'] = 'mosaic-bert-base-seqlen-2048'
+    json.dump(contents, open(config_file_path, 'w'), ensure_ascii=True)
+
+if __name__ == '__main__':
+    main()