Humphrey Sheil - Blog

Getting to grips with LSTM (part one)

14 Jan 2016 | 25 comments

What is LSTM (Long Short Term Memory)

Have you got a data set which you are sure contains some really interesting temporal relationships? Do you suspect that if only you could exploit this knowledge you could improve your target predictor / classifier? If yes, then you probably already know about LSTM as a potential solution to your problem.

LSTM (Long Short Term Memory) has a storied history. It dates back to 1997 so is not particularly new, yet it has only really come to prominence in the last two years (2013 - 2015) as:

  1. LSTM has usurped previous state of the art approaches in applications such as handwriting recognition, phoneme recognition and more.

  2. LSTM has replicated these successes in real-world applications (by contrast, many research ideas do not map across to industry use well, if at all).

I'm interested in LSTM for four reasons:

  1. As noted above, it represents current state of the art for a wide range of problem domains / applications. I have a hunch that there is something intrinsic to LSTM which can port to other problem domains and add value by improving classification / prediction accuracy.

  2. Ecommerce data has a strong temporal component - we denote web site users by some unique identifier (e.g. a cookie) and then we log their observable actions as a series of events on their timeline. Cause and effect is important, and we need something like LSTM to remember the important events and discard the less relevant examples. I work with Ecommerce data a lot, and not so much generating new sonnets or inception images :).

  3. LSTM is essentially deep learning where the depth is created by unrolling the network through time.

  4. LSTM is probably one of the main precursors to RAM (Reasoning, Attention, Memory) networks, and understanding / using LSTM can serve as a good base to apply RAM to target datasets. RAM is growing quickly..


Most of the data sets used in deep learning revolve around image and text corpuses. These sets and domains are very interesting, but they are too far away from ecommerce data to allow any useful conclusions or correlations to be drawn. So the goal for this series of posts is as follows:

  1. Build up a good understanding of the Torch7-based RNN framework by creating the simplest body of code which builds and trains an LSTM recurrent neural network.

  2. Use these simple building blocks to construct and train a model which consumes and classifies an Ecommerce data set with strong temporal connections embedded in it.

  3. Evaluate the model performance and apply well-known techniques to improve that initial performance.

RNN Framework selection criteria

What matters when picking / scoring an ML framework to use? Some criteria simply relate to ease of use and tooling support, while others speak directly to the ML ability of the framework. For me, the criteria are:

  • Good Language support (debugger, editor)
  • Speed of development
  • Active community
  • Good performance (training speed, data set scalability, accuracy / precision)
  • Near transparent migration from CPU to GPU training
  • Reproducible results
  • Ability to modify the framework / template code easily

The Torch + RNN layer we delve into here is by no means perfect, but it scores highly enough on these criteria to make it my current choice. Realistically, it pays to have good familiarity with multiple frameworks. TensorFlow (TF) for example may morph into some kind of generic back-end like gcc or clang where other ML frameworks output TF models for training. So there's no reason for me to move to TF until it demonstrates something superior - speed, ability to harness a cluster of nodes. And when it does, then I'll use it as well as Torch / RNN.

Other candidates

There are multiple frameworks available that "do" LSTM, for example:

I don't mind having to learn Lua or Python whereas a lot of researchers prefer the Python frameworks. I find the compile time overhead of Theano to be quite onerous so I prefer the interpreted nature of Torch7. Brainstorm is pretty new still and so is TensorFlow. I've played with Keras and it looks good, although I don't know that it lets me control enough directly.

Torch and the RNN library

I don't propose to cover Torch itself in this post - there is simply too much to cover. Suffice it to say that the core Torch distro does not support LSTM or recurrent neural networks (RNNs) natively. It is possible of course to construct RNNs / LSTM using the Torch nngraph (specifically the gModule). Three specific implementations that I m aware of that take this approach are:

  1. Wojciech Zaremba's LSTM code for the Penn TreeBank corpus.
  2. Andrej Karpathy's LSTM (char-rnn) code, built on Zaremba's code.
  3. Brendan Shillingford's tutorials for Nando de Freitas' ML course at Oxford.

I find this code hard to understand / change so I was looking for a better abstraction of LSTM cells with the various forget gates "already baked in". That is the primary thing that brought me to @nicholas-leonard's RNN addition to Torch.

The RNN library

Now, the interesting thing about the RNN library (and this in no small part prompted this sequence of posts) is that you don't need to use all of the RNN library to take advantage of LSTM. Therefore we will write the simplest code we can here and only add in the parts we need. One specific things we won't use just yet are:

  1. The dp package (Experiment, DataSource etc.). It's powerful, but confusing..

Structure of this tutorial

This tutorial builds as follows:

  1. Post 1 (this post) - setting out the groundwork and building the base code for future posts.

  2. Post 2 - Moving to use a real-world data set and to train on the GPU.

  3. Post 3 - Performance tuning - how to improve the performance of the model by applying tried and tested optimisations / tricks of the trade when (a) engineering features / input layers for neural networks and (b) tuning the model itself.

The core code

Ok, so our minimum requirements for the core code at this point are as follows:

  1. Create the requisite data structures so we can train an LSTM on a synthetic data set using RNN.
  2. Create an LSTM using the RNN library that is an adjunct to the main Torch repo.
  3. Train the LSTM.
  4. Save the trained LSTM and associated data structures to file so we know we can re-use trained models for prediction.
  5. Load in the saved model and data sets to validate all of our work so far.

Without further ado, here is the code to do exactly that! There's not really much point in building it up slowly / line by line - hopefully the inline comments will make it clear exactly what each non-obvious line is doing. Obvious lines are self-documenting :)

(Thanks to @nicholas-leonard for help in debugging this code as I wrote it)

require 'rnn'

cmd = torch.CmdLine()
cmd:text('Simple LSTM example for the RNN library')
cmd:option('-use_saved',false,'Use previously saved inputs and trained network instead of new')

-- parse input params
opt = cmd:parse(arg)

-- Keep the input layer small so the model trains / converges quickly while training
local inputSize = 10
-- Larger numbers here mean more complex problems can be solved, but can also over-fit. 256 works well for now
local hiddenSize = 256
-- We want the network to classify the inputs using a one-hot representation of the outputs
local outputSize = 3

-- the dataset size is the total number of examples we want to present to the LSTM 
local dsSize=200

--We present the dataset to the network in batches where batchSize << dsSize
local batchSize=5
--And seqLength is the length of each sequence, i.e. the number of "events" we want to pass to the LSTM
--to make up a single example. I'd like this to be dynamic ideally for the YOOCHOOSE dataset..
local seqLength=5
-- number of target classes or labels, needs to be the same as outputSize above
-- or we get the dreaded "ClassNLLCriterion.lua:46: Assertion `cur_target >= 0 && cur_target < n_classes' failed. "
local nClass = 3

function build_data()
   local inputs = {}
   local targets = {}
   --Use previously created and saved data
   if opt.use_saved then
      inputs = torch.load('training.t7')
      targets = torch.load('targets.t7')
      rnn = torch.load('trained-model.t7')
      for i = 1, dsSize do
         -- populate both tables to get ready for training
         table.insert(inputs, torch.randn(batchSize,inputSize))
         table.insert(targets, torch.LongTensor(batchSize):random(1,nClass))
   return inputs, targets

function build_network(inputSize, hiddenSize, outputSize)
   if opt.use_saved then
      rnn = torch.load('trained-model.t7')
      rnn = nn.Sequential() 
      :add(nn.Sequencer(nn.Linear(inputSize, hiddenSize))) 
      :add(nn.Sequencer(nn.LSTM(hiddenSize, hiddenSize)))
      :add(nn.Sequencer(nn.LSTM(hiddenSize, hiddenSize))) 
      :add(nn.Sequencer(nn.Linear(hiddenSize, outputSize))) 
   return rnn

function save(inputs, targets, rnn)
   -- Save out the tensors we created and the model itself so we can load it back in
   -- if -use_saved is set to true'training.t7', inputs)'targets.t7', targets)'trained-model.t7', rnn)

--two tables to hold the *full* dataset input and target tensors
local inputs, targets = build_data()
local rnn = build_network(inputSize, hiddenSize, outputSize)

-- Decorate the regular nn Criterion with a SequencerCriterion as this simplifies training quite a bit
-- SequencerCriterion requires tables as input, and this affects the code we have to write inside the training for loop
local seqC = nn.SequencerCriterion(nn.ClassNLLCriterion())

local start = torch.tic()

--Now let's train our network on the small, fake dataset we generated earlier
--Feed our LSTM the dsSize examples in total, broken into batchSize chunks
for numEpochs=0,200,1 do
   local start = torch.tic()
   for offset=1,dsSize,batchSize+seqLength do
      -- We need to get a subset (of size batchSize) of the inputs and targets tables
      local batchInputs = {}
      local batchTargets = {}

      --start needs to be "2" and end "batchSize-1" to correctly index
      --all of the examples in the "inputs" and "targets" tables
      for i = 2, batchSize+seqLength-1,1 do
         table.insert(batchInputs, inputs[offset+i])
         table.insert(batchTargets, targets[offset+i])
      out = rnn:forward(batchInputs)
      err = seqC:forward(out, batchTargets)
      gradOut = seqC:backward(out, batchTargets)
      rnn:backward(batchInputs, gradOut)
      --We update params at the end of each batch
   local currT = torch.toc(start)
   print('loss', err .. ' in ', currT .. ' s')
save(inputs, targets, rnn)

The code is also available here with better syntax highlighting.

Running and testing the code

On our first run, the code minimises the loss / error in about 200 epochs and this takes about 6 minutes. With a working torch installation, you can test this as follows:

time th example_part1.lua
loss      3.3337339206784 in        1.0516710281372 s 
loss      3.3362607493478 in        1.0598111152649 s 
loss      0.0011351864864528 in     1.1150119304657 s 
loss      0.0011246778926528 in     1.0118091106415 s 
loss      0.0011143412047407 in     1.12540102005 s   
loss      0.0011041739914276 in     1.0564870834351 s 
real    0m6.030s
user    0m7.156s
sys     0m1.202s

After you run the code you will see *.t7 files in the directory:

ls -laht *.t7
64 -rw-r--r--  1 hsheil  staff    31K 11 Jan 18:32 targets.t7
44288 -rw-r--r--  1 hsheil  staff    22M 11 Jan 18:32 trained-model.t7
216 -rw-r--r--  1 hsheil  staff   105K 11 Jan 18:32 training.t7

These files represent the trained model and inputs and targets - if we re-run the network again but load in these files, then the network should carry on training from where it left off:

>th example_part1.lua -use_saved
loss      0.0010941725196606 in     1.1108829975128 s 
loss      0.0010843310458659 in     1.1420600414276 s 
loss      0.0010746471778253 in     1.0338280200958 s

And it does! Note that we have simply over-trained our LSTM on the random problem - but it's hard at this point to get it to do well on a similarly generated random validation or test set. That will have to wait until part two, but we now have the plumbing code in place to perform our cross-validation in part two.

Part two..

In the next part of this series of posts, we will:

  1. Replace the play / synthetic data set with a real data set (this will expose a lot of "in the trenches" coding that is needed to make LSTM work), including having a separate validation and test set.

  2. Add code so we can target dedicated hardware (efficiently) to speed up training - using CUDA and OpenCL. We can expect a 5x - 15x speedup running on GPUs, depending on how well we can keep the GPU fed with data without context-switching between it and the CPU.


11 Aug 2017

serial code the sims 2 apartment life


13 Jul 2017

Наиболее полезные гаджеты для авто Даже если у вас не новый автомобиль, вы довольно просто можете увеличить его возможности, с применением полезных гаджетов, приобретенных в популярном онлайн-магазине в котором есть все для автомобилей и не только!   В наше время, когда дорожно-транспортных происшествий на центральных дорогах нашего государства становиться все больше, полезной вещью в машине считается авто регистратор. Это техническое устройство, заметно более полезное чем <a href=>купить миксер дешево </a> , сможет зафиксировать на флэш-карту каждую минуточку вашего нахождения за рулем, а в случае аварии видеозапись можно использовать в качестве доказательства правоты


01 Jul 2017

301 Moved Permanently <a href=>More info>>></a>


12 Jun 2017

Краткий обзор наиболее интересных MMO игр 2017 года В приведенной ниже статье мы вкратце рассмотрим для вас группу самых многообещающих MMO игр от передовых издателей нашей планеты например про mmorpg онлайн игры , дабы вы сумели отыскать себе игру по-нраву, которая поможет вам скоротать не один долгий зимний день. Обзоры и все о новинках игр ММО на сайте -


07 Jun 2017

Для каких целей геймеры применяют трейнеры и подробные гайды к online играм Гайды, которые регулярно публикуются на портале сегодня считаются единственным инструментом, позволяющим заметно облегчить себе трудный процесс прохождения популярных online-игр. Дело в том, что солидная часть процессов и квестов в онлайн играх завязана на сервера разработчиков, и потому использовать читы и трейнеры в них попросту нереально. Гайд для современной игры — это небольшая инструкция, руководствуясь которой вы сумеете достигнуть важной цели с минимумом затрат с вашей стороны. Чтобы увидеть, как выглядят хорошие гайды, прочитайте материал <a href="">игры онлайн бесплатно новинки</a> .


04 Jun 2017

В какие online RPG можно поиграть в 2017 году ? Люди, которые в раннем подростковом возрасте безмерно обожали многопользовательские онлайн игрушки, к сорока пяти годам в большинстве случаев отстают от моды и не знают, в какие компьютерные игры надо играть в наши дни. Данная публикация была написана для взрослых людей, которые решили вспомнить юношеские годы и поиграть в качественную браузерную игрушку. Давайте обсудим группу самых модных проектов! <a href="">браузерные стратегии</a> , позволит вам уже через парочку минут приступить к развитию своего персонажа и отправиться на битву с другими игроками в PvP режиме.


30 May 2017

Новая веха в экономике и машиностроении началась с созданием достаточно эффективных двигателей внутреннего сгорания, которые стали устанавливать во множество машин, в том числе и в сель трактора. Именно благодаря изобретению такого надежного двигателя в наши дни вы можете купить себе <a href=>отвал бульдозера б170</a> . Очередным полезным усовершенствованием для тракторов может считаться изобретение гусениц, как пишет сайт, они позволили в десять раз увеличить мощность трактора и положили начало их эксплуатации в странах с не самым благоприятным климатом. Когда на полях появились такие трактора, сельское хозяйство очень быстро перестало нуждаться в таком большом объеме ручного труда. И поэтому миллионы граждан освободились и были вынуждены искать себе иную сферу занятости. И нередко такими местами являлись большие фабрики и заводы, что начали массово строить во многих странах.


29 May 2017

Советских времён ванные, отлитые из уральского чугуна, всегда могли похвастаться своей прочностью и долговечностью. И в большом числе киевских домов они служат и в наше время! Однако даже этим удивительным изделиям с течением лет свойственно разрушаться и терять свой начальный лоск: идеально гладкая эмаль на них покрывается сеткой трещин и становится желток. К счастью утратившую товарный вид чугунную ванную можно восстановить с помощью акрила, подробно данный метод рассмотрен на ресурсе ванны.укр/блог/ремонт/наливные-полы-3д-своими-руками.html . Если восстановлением старой ванны занимаются специалисты своего дела, то полученное в конце поделие сможет прослужить вам еще минимум пятнадцать лет.


02 Mar 2017

<a href=>Бильярд онлайн</a> бесплатно, играй и получай удовольствие. Бильярд онлайн это потрясающая игра которая подойдет абсолютно всем независимо от возраста.


02 Mar 2017

<a href=>Бильярд онлайн</a> бесплатно, играй и получай удовольствие. Бильярд онлайн это потрясающая игра которая подойдет абсолютно всем независимо от возраста.


27 Feb 2017

Zhheherhrh eg egemtr hrggwe wes egemtrghdfbsgd rwtwrqw


26 Jan 2017

CroatiaI am a big fashion fan and pretty much everything in my life was about shoes, bags, jackets (really love th.) month and a half ago, I become a mother of a baby boy and I just cannot lie and say that I would take a picture of anything else but his first smile or his amazing pettit hands.He is my most precious inspiration and my life.But I will say ...that boynis going to have style:-) kiss


26 Jan 2017

Scott Wulff / Howard Aubrey Well, Mr. Aubrey, you show your real ignorance in this case! I am not morbidly obese and I have to agree with Mr. Rivera. The real reason these knee jerk reactions are happening is because the goenevmrnt thinks that people are too stupid to make up their own minds and that they know what is good for everyone.


13 Nov 2016

Ugireojfe whfiwehfjwehwhfjehfwefhweh 777uiop fweh iwehf weiohf wieohf iwehf iweyu59tu328hfire iuwfodhqw934785 h3urh9wjfwgut h9wh9889wh98r h4wt93qrj29th2 rj2ghw9tfq.


13 Nov 2016

Xighefjeo orj wokwp dkow pwk wodj d hfdgfhgf 4756 5uhtyjur urt45


11 Nov 2016

Pgksrjgiohi hw hweokfjeq ojfe jfweiogwo gwoj wijf gdhgtrj575 y6u75tyhgf 5yu5regr


10 Nov 2016

Xighefjeo orj wokwp dkow pwk wodj d hfdgfhgf 4756 5uhtyjur urt45


08 Nov 2016

Ugireojfe whfiwehfjwehwhfjehfwefhweh 777uiop fweh iwehf weiohf wieohf iwehf iweyu59tu328hfire iuwfodhqw934785 h3urh9wjfwgut h9wh9889wh98r h4wt93qrj29th2 rj2ghw9tfq.


06 Nov 2016

Pgksrjgiohi hw hweokfjeq ojfe jfweiogwo gwoj wijf gdhgtrj575 y6u75tyhgf 5yu5regr


05 Nov 2016

Ugireojfe whfiwehfjwehwhfjehfwefhweh 777uiop fweh iwehf weiohf wieohf iwehf iweyu59tu328hfire iuwfodhqw934785 h3urh9wjfwgut h9wh9889wh98r h4wt93qrj29th2 rj2ghw9tfq.


05 Nov 2016

Xighefjeo orj wokwp dkow pwk wodj d hfdgfhgf 4756 5uhtyjur urt45


04 Nov 2016

Pgksrjgiohi hw hweokfjeq ojfe jfweiogwo gwoj wijf gdhgtrj575 y6u75tyhgf 5yu5regr


08 Oct 2016

I'm so glad I found my soolitun online.


14 Jul 2016

Hi, I have two questions regarding first part: 1. We've created data into batches already, then, why do we do it again during training ? (from 114 th line of code) 2. we're setting an offset, I think it starts from 3rd position. Why? (from 121 th line of code)

Show more

Leave a comment

  Back to Blog