Training Neural Network on Large Datasets

9 visualizaciones (últimos 30 días)
Christopher
Christopher el 3 de Sept. de 2014
Comentada: Greg Heath el 9 de Jun. de 2018
Hello, I have a huge dataset that I am training a feedforward neural network on. This causes out of memory errors during training (on a huge server) even when I switch from trainlm to trainrp.
I have been reading about the adapt function for incremental learning, but I don't think that it's the way to go as many people say it gives bad performance and it's more appropriate when the nature of the data changes over time.
One idea I had was breaking the training data into smaller portions and running the train() function on each smaller portion and passing the neural network back into it each time. However, I am concerned that train will lose the previous training and start training the network afresh on each new portion.
What is the best way to train this network on all this data?
  3 comentarios
hossam eldeen
hossam eldeen el 17 de Sept. de 2017
Editada: hossam eldeen el 17 de Sept. de 2017
Dear Greg Heath I am having the same problem. huge training data to be used in training. and i am always out of memory. i am using 2 giga ram,the input vectors are 200*800000 and the output is 5*800000. what can i do ? the purpose is classification
Greg Heath
Greg Heath el 6 de Jun. de 2018
You have 5 classes. I assume the target columns are from eye(5).
The input dimensionality is 200. The first task is to reduce that dimensionality.
Typically, ~35 random samples per dimension are more than adequate to define each class. Accordingly, ~35,000 random samples should be adequate to define the 5 classes in a 200-D space.
At this point you might think about reducing the dimensionality using PLS (PCA may not work well for classification).
Although this is just a start, I expect this to point you in the right direction.
Greg
Let us know if you run into roadblocks.
Greg

Iniciar sesión para comentar.

Respuestas (1)

Greg Heath
Greg Heath el 17 de Sept. de 2017
YOU ARE IN LUCK!!!
1. Randomly divide each of the c classes into s subsets.
2. Randomly combine one subset from each class and design s c-class neural net subclassifiers
3. Design a combiner that takes the outputs of the s subclassifiers and outputs a final class index.
I was the first to do this for U.S. Army radar target classification in the early 1980s.
Have fun!
Greg
  4 comentarios
H.k
H.k el 9 de Jun. de 2018
Editada: H.k el 9 de Jun. de 2018
I tried to divided data in my code but i still having a problem please mr,Grig take a look on the below code I don't know why it gives me the same output
x=rand(720,2721);
y=rand(720,1);
net=fitnet(10);
Nets=[];
for i=1:720
P=x(i,:)';
T=y(i,:)';
net=train(net,P,T)
Nets=[Nets,{net}]
end
% at simulating results
h=[];
for i=1:720
net=Nets{i}
a=net(P)
h=[h,{a}]
end
%the variable h contains constant value
Greg Heath
Greg Heath el 9 de Jun. de 2018
This thread is a standard for confusion. Multiple authors on multiple topics (e.g., classification AND regression!)
1. You definitely should have started a new thread!
2. It is ridiculous to use an example that has both random inputs and random targets!!!
3. What you are doing makes no sense at all. You are training a net for each input???
4. Go back and look at the documentation examples
help fitnet doc fitnet.
5. Start a new question.
Hope this helps
Greg

Iniciar sesión para comentar.

Categorías

Más información sobre Image Data Workflows en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by