ELECTRA
A couple weeks ago, I released a couple models in huggingface/transformers.
Just put up two new models in @huggingface transformers!One is a Bert-based unofficial implementation of ELECTRA-small, and the other is an Albert model (calling it ALECTRA, very creative) created with the same pre-training task 1/4 pic.twitter.com/yuwJQySLYv— Sho...
Different levels of collate_fn
Some tasks require to provide many objects per data point.
To handle creating batches of these complex data points, pytorch lets you write
a collate_fn (official documentation).
This is really useful if you’re trying to perform a task like BERT training:
encoder_input_ids, encoder_mask, decoder_input_ids, decoder_mask, token_type_ids = batch
...