A couple weeks ago, I released a couple models in huggingface/transformers. Just put up two new models in @huggingface transformers!One is a Bert-based unofficial implementation of ELECTRA-small, and the other is an Albert model (calling it ALECTRA, very creative) created with the same pre-training task 1/4 pic.twitter.com/yuwJQySLYv— Sho...

Read more

Different levels of collate_fn

Some tasks require to provide many objects per data point. To handle creating batches of these complex data points, pytorch lets you write a collate_fn (official documentation). This is really useful if you’re trying to perform a task like BERT training: encoder_input_ids, encoder_mask, decoder_input_ids, decoder_mask, token_type_ids = batch ...

Read more