[D] Generative Model FOr Facts Extraction

Submitted by Zetsu-Eiyu-O t3_10q45pr in MachineLearning

Is it possible to finetune a generative model (like T5) to do something like this:

{

inputs: "XYZ <eot> XYZ was born in ABC. They now live in DEF.",

targets: "XYZ <t> born in <t> ABC <f> XYZ <t> lives in <t> DEF"

}

Like the transformer model fom this paper

if so how should I go about approaching the problem?

Is this task as simple as feeding it the inputs and targets or do you guys think it has more to it?

1

Comments

You must log in or register to comment.

MysteryInc152 t1_j6o62z6 wrote

This is what In-context learning is for.

Giving the model a few examples of a text input and a corresponding fact extraction is all that's necessary.

0

Zetsu-Eiyu-O OP t1_j6o9sz5 wrote

so just tokenize the output and penalize it against that? so do all generative models (for example gpt2) can learn this output the same way?

1

MysteryInc152 t1_j6okowf wrote

Not sure what you mean by penalize but say you wanted an LLM that wasn't instruction fine-tuned to translate between 2 languages it knows.

Your input would be

Language x: "text of language x"

Language y: "translated language x text"

You'd do this for a few examples. 2 or 3 should be good. Or even one depending on the task. Then finally

Language x: "text you want translated"

Language y: The model would translate the text and output here

All transformer generative LLMs work the same way with enough scale. GPT-2 (only 1.5b parameters) does not have the necessary scale.

1

Zetsu-Eiyu-O OP t1_j6oztfw wrote

Oh, I see thanks, I have a few questions about the basics of training a large language model, do you mind if I shoot you a message?

1