ahiddenmessi2 OP t1_jad9upi wrote on February 28, 2023 at 4:41 PM

Reply to comment by I_will_delete_myself in [D] Training transformer on RTX2060 by ahiddenmessi2

Thank you. I am looking at codeBERT which might satisfy my needs

ahiddenmessi2 OP t1_jaciwqg wrote on February 28, 2023 at 1:31 PM

Reply to comment by ggf31416 in [D] Training transformer on RTX2060 by ahiddenmessi2

Thanks for your reply. My goal is to train the transformer to read a specific programming language so I I guess there is no pre trained model available. Seems I have to train it from scratch on my laptop GPU :(

Edit: and yes it has 6gb only

ahiddenmessi2 OP t1_jacin2n wrote on February 28, 2023 at 1:29 PM

Reply to comment by KingsmanVince in [D] Training transformer on RTX2060 by ahiddenmessi2

My dataset size can be varied cuz the data can be generated. Also, I will consider using gradient accumulation to improve performance too. Thanks

ahiddenmessi2 OP t1_jachs8r wrote on February 28, 2023 at 1:22 PM

Reply to comment by aigoritma-1 in [D] Training transformer on RTX2060 by ahiddenmessi2

Thank you I will look into it

ahiddenmessi2 OP t1_jacahig wrote on February 28, 2023 at 12:10 PM

Reply to comment by CKtalon in [D] Training transformer on RTX2060 by ahiddenmessi2

Thank you . I will take a look of my number of parameters .