ideas
ideas copied to clipboard
Machine learning embedding middleware
Project description
There are lot of neural networks focused on doing things similar to machine translation, but not limited to text:
- transforming text into sound of speech
- transcribing text into speech
- describing pictures with text
- drawing pictures based on text descriptions
- changing voice
- replacing faces
- drawing faces based on principal vectors in latent space
- machine translation from one language into another
- creating HTML code basing on a picture drawn by designer
- picture style transfer
They usually work the following way:
1 encoder network encodes initial representation into a vector in latent space 2 decoder network decodes that feature vector into the things needed
Encoders and decoders are trained simultaneously and differ from a model to model, so do the internal representations.
The idea is to take as many different nets as possible, cut in the middle where the internal representations appear, then add some small net to each (let's call it a transcoder), connect different nets and train the transcoders to converge to the same representations. Then standardize the internal vector meaning. Then we should be able to use combinations of sources and sinks available in no individual model.
Notation
Let's call the tyes of content on the ends of a network modalities
:
-
seq<char, en>
(a seq of characters, forming a text in English) -
seq<char, ru>
-
seq<word_vec<...>, en>
-
picture
- sound features
- source code features
- face features
Let's call intermediate representations transcoding modalities
.
Let's call a model tranforming between transcoding modalities a transcoder
.
In the following text we are using oriented graphs. An edge A -> [M] -> B
means that model
(under model here we mean that we can compute gradients) M
transforms data in modality A
into modality B
. An edge A => [O] => B
means it is possible to convert data in modality A
into modality B
using a black-box oracle O
, such as a TTS program or external API. We may call an edge a model
or an oracle
.
API
ModalityId
is an identifier of the modality. The standardized global representation has identifier global
. Then there are modality-specific representations, which names are also standardized, like speech_latent
, image_latent
, etc...
When installed, the middleware exposes the API to applications:
getInfo(out info)
- returns information about installed models, oracles and their modalities available.
startTransaction(out txId)
- starts a transaction. All the models adding/removing stuff is done within transactions. In the end of a transaction the middleware checks consistency and removes all the transcoders ending to unreferenced transcoding modalities.
commit(in txId, out error)
- checks consistency and commits
registerModel(in our_model_id, in inModalityId, in outModalityId, in our_model)
adds a net into the registry. our_net_id
is the globally unique net id. Transcoder networks are registered the same way as modalities networks, but for ids uuids are used. When a uuid is used, the server recognizes the modality as a transcoding modality
.
ourModel
is the model serialezed, for example, into ONNX or CNTK.
unregisterModel(in netId)
- removes the model.
registerModalityType(in name, in parameterCount)
registerModality(in type, in arguments)
registerDatasetType(in name, in modalityType, in modalityType)
registerOracleType(in sourceModalityType, in targetModalityType)
registerOracle(in oracleType, in entryPoint)
unregisterOracle(in oracleId)
getPath( in inputMkdalityId, in outModalityId, in pieces, in preference, out path)
- finds the optimal path
(a sequence of models) between modalities, using pieces
to require certain networks and representations to be in a path and using preference
as criteria of optimality.
convert(in inputRepresentation, in inputModalityId, in outModalityId, in path, in returnGradients, out outputRepresentation)
- converts the input into output. If returnGradients
is true
, gradients are returned.
getModel(in netId, out net)
- returns the serialized model. Used for training transcoders.
getTrainingOpportunities(in inModality, in outModality, out trainingOpportunities)
- returns training opportunities.
Training opportunities
In order to train a model one needs a dataset. Datasets can be obtained different ways. getTrainingOpportunities
is a function analysing the graph of available models and oracles and returning the candidates usefult for trining the specific model betaeen 2 modalities.
If G is the graph and we want to train a model A -> [M] -> B
, where A and B are subgraphs in G, then a suitable dataset a, b
(where a
and b
are modalities) is one, satysfying the following conditions:
- a \in V_A, b \in V_B
- for all b'', b' \in V_B \exists E_{b'', b'}, which is a
model
.
Then the middleware matches the endpoints against abailable dataset types.
A TrainingOpportunity
is an object containing leaning graph and datasets matching this graph.
Training software should use training opportinities in order to get the needed datasets (for example from a repository of datasets like OpenML or using a database instructing the software how to retrieve and preprocess datasets or asking a user to do that) and then train the needed model.
Example
Assume we have the models installed (all models names are random)
-
text<en> -> [Painter] -> [PainterTranscoderText] -> text_latent <-> global <-> image_latent -> [PainterTranscoderImage] -> [Painter] -> image
-
speech<ru> -> [RuSpeechRecog] -> [RuSpeechRecogTranscoder] -> speech_latent <-> global <-> text_latent -> [RuSpeechRecogTranscoder] -> [RuSpeechRecog] -> text<ru>
And oracles registered:
-
text<α> => [TTS] => speech<α>
-
text<α> => [machine translation] => text<β>
And datasets types registered:
-
text<α>
-text_corpuses<α>
(Wikipedia, Gutenberg, texts in Internet) -
text<α>, text<b>
-bilingval_sources<α, β>
-
speech<α>
-speech corpuses<α>
(videohostings, podcasts, songs) -
speech<α>, text<α>
-transcribed<α>
videos and audios
And there exists a pretrained model text<en> -> [EnTTS] -> {..EnTTSLatent..} -> {..EnTTSLatent..} -> [EnTTS] -> speech<en>
.
We wanna create 2 transcoders
-
{..EnTTSLatent..} -> [EnTTSTextTranscoder] -> text_latent
-
speech_latent -> [EnTTSSpeechTranscoder] -> {..EnTTSLatent..}
So for the first transcoder we call
trainingOpportunities = getTrainingOpportunities("{..EnTTSLatent..}" , "text_latent")
and should get
-
text<en> -> [EnTTS] -> {..EnTTSLatent..} -> [EnTTSTextTranscoder] -> text_latent <- [Painter] <- text<en>
(text_corpuses<en>
,bilingval_sources<en, β>
,transcribed<en>
) -
text<ru> => [machine translation] => text<en> -> [EnTTS] -> {..EnTTSLatent..} -> [EnTTSTextTranscoder] -> text_latent -> [RuSpeechRecogTranscoder] -> [RuSpeechRecog] -> text<ru>
(text_corpuses<ru>
,bilingval_sources<ru, β>
,transcribed<ru>
) -
text<en> -> [EnTTS] -> {..EnTTSLatent..} -> [EnTTSTextTranscoder] -> text_latent -> [RuSpeechRecogTranscoder] -> [RuSpeechRecog] -> text<ru>
(bilingval_sources<en, ru>
) -
text<ru> => [machine translation] => text<en> -> [EnTTS] -> {..EnTTSLatent..} -> [EnTTSTextTranscoder] -> text_latent <- [Painter] <- text<en>
(bilingval_sources<en, ru>
)
Example usage
Text to speech
path = server.getPath("text<en>", "voice<en>")
hwSound = server.convert("hello world", path)
... # play sound
Creating a transcoder
... #load model
... #cut it in the middle: EnTTSText , EnTTSSpeech
txId=server.startTransaction()
server.registerModel("EnTTS", "text<en>", "{..EnTTSLatent..}", EnTTSText)
server.registerModel("EnTTSTextTranscoder", "{..EnTTSLatent..}", "text_latent", None) # model is yet to be trained
server.registerModel("EnTTSSpeechTranscoder", "speech_latent", "{..EnTTSLatent..}", None) # model is yet to be trained
server.registerModel("EnTTS", "{...}", "speech<en>", EnTTSSpeech)
server.commit(txId)
def train(m1, m2, hyperparams):
trainingOpportunities = server.getTrainingOpportunities(m1, m2)
metatrainer = MetaTrainer( m1, m2, hyperparams)
def transformPath(p, isSecondPart):
for link in p:
if isinstance(link, ModelLink):
bigNet.stack(getModel(link.modelId), frozen=True)
elif isinstance(link, OracleLink):
metatrainer.transformedData[link.dstModality] = link.oracle.transform(metatrainer.transformedData[link.srcModality])
def transformPart(ds, m1, isSecondPart):
for m in ds:
if not isSecondPart:
p = findPath(m, m1)
else:
p = findPath(m1, m)
if not p:
continue
metatrainer.transformedData[m] = ds[m]
transformPath(p, isSecondPart )
for op in trainingOpportunities:
retriever.retrieve(op.datasets)
for ds in op.datasets:
bigNet = metatrainer.createFlow()
transformPart(ds, m1, False)
bigNet.stack( metatrainer.core, frozen=False)
transformPart(ds, m2, True)
metatrainer.train()
return metatrainer.core
trained=train("{..EnTTSLatent..}" , "text_latent", ....)
txId=server.startTransaction()
server.registerModel("EnTTSTextTranscoder", "{..EnTTSLatent..}", "text_latent", trained)
server.commit(txId)
trained=train( "speech_latent", "{..EnTTSLatent..}" , ....)
txId=server.startTransaction()
server.registerModel("EnTTSSpeechTranscoder", "speech_latent", "{..EnTTSLatent..}", trained)
server.commit(txId)
Complexity and required time
Complexity
- [x] Advanced - The project requires the user to have a good understanding of all components of the project to contribute
Required time (ETA)
- [x] Much work - The project will take more than a couple of weeks and serious planning is required
Well this idea is great, now the problem is collecting data, what about making project based around the collection of data
Well this idea is great
I am not sure. I am not a NN pro, maybe the idea is useless junk.
now the problem is collecting data, what about making project based around the collection of data
We don't need to train models from scratch. It is proposed to use pretrained models, cut them in the middle, and add small adapter networks. They are not as large as full networks, so they don't need so much data for training.