neuralconvo.ini + missing data/train.enc

[johndpope]

the neuralconvo.ini specifies following files

[strings]

Mode : train, test, serve

mode = train
train_enc = data/train.enc
train_dec = data/train.dec
test_enc = data/test.enc
test_dec = data/test.enc

but there is no data folder in repo.
there is the working_dir

python3 execute.py

Mode : train

Preparing data in working_dir/
Tokenizing data in data/train.enc
Traceback (most recent call last):
File "execute.py", line 313, in
train()
File "execute.py", line 127, in train
enc_train, dec_train, enc_dev, dec_dev, _, _ = data_utils.prepare_custom_data(gConfig['working_directory'],gConfig['train_enc'],gConfig['train_dec'],gConfig['test_enc'],gConfig['test_dec'],gConfig['enc_vocab_size'],gConfig['dec_vocab_size'])
File "/Users/johndpope/Documents/gitWorkspace/tensorflow_chatbot/data_utils.py", line 137, in prepare_custom_data
data_to_token_ids(train_enc, enc_train_ids_path, enc_vocab_path, tokenizer)
File "/Users/johndpope/Documents/gitWorkspace/tensorflow_chatbot/data_utils.py", line 121, in data_to_token_ids
normalize_digits)
File "/Users/johndpope/Documents/gitWorkspace/tensorflow_chatbot/data_utils.py", line 100, in sentence_to_token_ids
words = basic_tokenizer(sentence)
File "/Users/johndpope/Documents/gitWorkspace/tensorflow_chatbot/data_utils.py", line 50, in basic_tokenizer
words.extend(re.split(_WORD_SPLIT, space_separated_fragment))
File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/re.py", line 203, in split
return _compile(pattern, flags).split(string, maxsplit)
TypeError: cannot use a bytes pattern on a string-like object

[Fogapod]

[jekriske]

The gitignore excluded his data folder from being checked in.

[jprissi]

How do you create the .enc files from the movie-dialogs corpus he put in the readme ?

EDIT : Okay got them, https://github.com/suriyadeepan/datasets/tree/master/seq2seq/cornell_movie_corpus/

[jekriske]

doh, can't perform git-lfs check for the repo properly because the account is over quota:
"This repository is over its data quota. Purchase more data packs to restore access"

[drewp]

To rebuild them:
mkdir tensorflow_chatbot/data
cd tensorflow_chatbot/data
Get https://people.mpi-sws.org/~cristian/data/cornell_movie_dialogs_corpus.zip, put the *.txt files in this new data/ dir.
git clone https://github.com/suriyadeepan/datasets.git
Edit datasets-master/seq2seq/cornell_movie_corpus/scripts/prepare_data.py and uncomment the last lines so prepare_seq2seq_files executes.
python datasets-master/seq2seq/cornell_movie_corpus/scripts/prepare_data.py
This makes {train,test}.{enc,dec}

These files are just lines of text. I guess matching line numbers between enc and dec are conversation pairs.

[jekriske]

I had to grab that script manually as I couldn't checkout the repo as mentioned. It appears to be training now.

[jprissi]

Same here, had to correct a few issues with python 3.5 and the use of re.split that caused errors.

[shlomis]

What was the re.split issue?

[Niko2756]

it is a problem with this line

line 8, in get_id2line:
lines=open("movie_lines.txt").read().split('\n')

it gives this error and will not let the file execute:
line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 585399: invalid start byte

how did you guys fix it?

[Niko2756]

got it you have to run the prepare_data.py in python 2.7

you have to also uncomment the last lines so prepare_seq2seq_files executes.

and make sure you have a python 2.7 environment for the prepare_data.py and remember the assignment code is in python 2.7 as well.

[jprissi]

@shlomis The re.split issue was about 're' not able to apply a byte rule to a string while training. Had to replace the line 50 of data_utils.py :
words.extend(re.split(_WORD_SPLIT, space_separated_fragment))
with
try: words.extend(re.split(_WORD_SPLIT, str.encode(space_separated_fragment))) except: words.extend(re.split(_WORD_SPLIT, space_separated_fragment))
as if you just .encode the string you'll get the reverse error while testing

[jprissi]

@Niko2756 I am able to run it on Python 3 with a minimum amount of tweaks (a few print to change and maybe a small error message)

[johndpope]

I think this could be related.
suriyadeepan/datasets#1