Network design for a network learning sentences

[Brajesh]

Hello,

I’ve done a little bit of reading from the primers recommended.
I'm ready to get my feet wet!

I want to design a network that takes a sentence as an input, and outputs a sentence based on the input sentence.  

I want to use the book “Pride and Prejudice” as my training text.  I have the text from project gutenberg.

I calculated that the text has 6300 unique words.
I haven’t calculated yet, but I think the longest sentence might be around 100 words.

So I’m thinking of just mapping the words and numbering them from (1-6300) as inputs.
I’m also thinking I would need a recurrent network.

The intention for this project is to have fun, learn Simbrain, get my feet wet.
To see how well the network learns the prediction of sentences.  
And also how it responds to novel sentences.

How would you recommend my network design should look like to accomplish this?

Thank you

[jyoshimi]

Hi Brajesh,

I'm busy at the moment, hence the delay.  Here are some very quick thoughts.

First, for a fun discussion of training recurrent networks on textual data, see this

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

I think it's one of my favorite blog posts on any topic of all time!  Check out the fake-math!  And search for Tolstoy.   Karpathy trained a network to produce fake Tolstoy and he shows network output at different phrases of training.   It takes some effort but I was able to get the system he provides running and created some fake Nietzsche. You should be able to get it to work on Pride and Prejudice. Someone else used similar methods to make a fake movie

https://www.youtube.com/watch?v=LY7x2Ihqjmc

I'll be interested to see how far you get in Simbrain.   To start, I suggest just getting familiar with the text world interface. It's pretty minimal at the moment, but it will get you started seeing what happens when text gets converted to numbers.   You could try to make a modified Elman network that predicts the next sentence of Austin's novel but I think it will be hard to do and slow.   For something that will actually run and do something that might be fun to watch, I'd suggest just making a reader world for the Austin text, loading it with a dictionary created from maybe 200-1000 words of the 6500, linking it to a recurrent network, then having that feed back to a display world using the same dictionary.   So it  will be Austin in, and Austin-words out.   Then just mess with the network in the middle.  

I have some new youtube videos I still need to edit and post, and I am planning to make one about the text world stuff as well....Maybe I'll demo what I described above...

-  Jeff    

[Brajesh]

Ha ha the video is hilarious and well produced !

Yes please do the demo for the Nietzsche !!!

The article looks fascinating, I'm looking forward to reading and digesting it, and I'm also get familiar with the text world interface.

Thank you :)))

[Brajesh]

Jeff,

I read up on the text world and display world but I don't think I have enough information in the help files to do this.  Please will you give me some steps on how to do this.

I did "insert->new world-> text world reader"
Do I just paste the unique words into this file?
Then insert a new SRN?
Then open the couplings manager and connect the dictionary to the SRN?
Why only 200-1000 of the dictionary words, why not all 6500?
The dictionary is just used for mapping right word to a vector or scalar right?  It's not the training data?
Do you recommend using vector or scalar for my dictionary?

Pardon my ignorance,

thank you

[jyoshimi]

Hi again Brajesh,

I continue to be pretty busy but the good news is that part of what I'm busy with is Simbrain.   We were working on 4.0 but that is such a big undertaking and the release is so far off in the future that we have started shifting to another 3.x release, right now 3.03.   And there will probably be a 3.04 and 3.05 after that.   One of the things we can try to improve a bit in that time is text world.  It will have to wait until 4.0 for us to do any kind of real serious restructuring of the text stuff, but we can improve what we have until then.

The dictionaries are used to map words either to scalars or vectors, and then to map scalars and vectors back to words.   See these doc pages for an explanation

http://www.simbrain.net/Documentation/docs/Pages/Worlds/TextWorld/ReaderWorld.html

http://www.simbrain.net/Documentation/docs/Pages/Worlds/TextWorld/DisplayWorld.html

I'm not sure what the 200-1000 issue you ran in to was. Did you come across some limit on how many items you could add to a dictionary?

The dictionaries have a weird and kind of silly format now.  I might change that in a 3.x release.  But probably the easiest way to make a dictionary now is to manually create it using the csv format.

Here is one thing you can do to just get started.  

1) Create a recurrent network.  Open a network, add a neuron group, and connect it to itself with a synapse group.  This is described in the youtube video on building networks in Simbrain.
2) Open up a display world
3) Select Edit > Extract dictionary in the display world
4) Open the coupling manager
5) Set the left panel in the display world  to the network and select a bunch of neuron producers, and set the right panel in the coupling manager to the display world, and select a bunch of the scalar word consumers.  Then click add couplings.  

Now you can make a neural network create speech using words from Jane Austin.   Then you can change the sparsity and other structures of your recurrent network to modify what that speech looks like.

This is not nearly what you are looking for, but it's something.   If you have suggestions on improving reader and display worlds let me know and we can try to improve them a bit in 3.03 or 3.04, like I said.

I know I'm not answering all your questions.  Feel free to ask followups, but I will be pretty busy for a few weeks and so it might take me a while to respond and I'm not sure how far I can get you towards your goal.   But I'll do my best.  

- Jeff

[Brajesh]

Hi Jeff,

I'm confused on the dictionaries.

Do both the reader and display world refer to the same dictionaries i.e the "token dictionary" and "vector dictionary"?

Where are "cheese, flower, mouse, poison, yuck!, yum!" stored, ie. in which file?  
I try to delete them but they keep re-appearing?

thank you

[Brajesh]

Hi Jeff,

I built the above but I mostly don't know what I'm doing.
I don't know how to create the training data.  
I'm not sure if the reader world will take my text words and convert them to training input for the RN?

I was looking at the elmanSentences script.
I think there are similarities to this network and what I'm trying to do, it seems to be using a reader and display world with dictionaries etc.

Do you have any documentation on how the elmanSentences network was built?

thank you

[jyoshimi]

Yeah that was just so you could see something happen using your text, but there is no learning.   To do something like Elman Sentences you'd have to edit the script directly.  The things done in that script can't be done using the GUI alone, so you have to modify a script.  There is no real documentation, but you can ask me about specific lines if you like.   In 3.03 there will be an alternative to scripting where you use an IDE like eclipse which is easier.  Wish I could be of more help.

[Brajesh]

Oh!  
I just realized that there is elmanSentences.bsh file !

While it appeared from my previous posts that I knew that, I really didn't, I was just referring to the menu item "Scripts" and I had no idea that a script underneath was building the network.

No wonder I couldn't figure out how you were doing it from the GUI.

Ok great,
I'm going to read the script and see if I can understand what's going on and I'll be back with any questions,

thank you!

[jyoshimi]

Oh good, a bit of progress then!