spark out of memory error

Modérateur : Crucianu

Répondre
Min
Messages : 9
Inscription : 25 mars 2015 15:31

spark out of memory error

Message par Min » 31 mai 2015 19:14

Bonjour,

Je souhaite calculer l'ACP d'une matrice de 1500*10000. J'ai alloué 8g de mémoire (driver-memory=8g). Cependant j'ai l'erreur de out of memory. J'ai vu que la memory store est à 3.1g.

J'ai vu sur le site de spark que "spark.storage.memoryFraction" est défini à 0.6.

Voici mes questions:
1. Pouvez vous confirmer qu'en réalité c'est 3.1g qui est allouée? J'ai vu que la taille de block ne dépasse meme pas 1G, comment ca se fait que 3.1G n'est pas suffisant?
2. Est-ce que je peux modifier cette memory store que je suppose qu'il s'aigt de 0.5 de driver memory? Si oui, comment?

Merci par avance.

Voici une partie des logs:
15/05/31 16:17:02 INFO BlockManager: Found block rdd_3_0 locally
15/05/31 16:17:02 INFO BlockManager: Found block rdd_3_1 locally
15/05/31 16:23:00 INFO MemoryStore: ensureFreeSpace(401995375) called with curMem=539472846, maxMem=4445763010
15/05/31 16:23:00 INFO MemoryStore: Block taskresult_5 stored as bytes in memory (estimated size 383.4 MB, free 3.3 GB)
15/05/31 16:23:00 INFO MemoryStore: ensureFreeSpace(401995375) called with curMem=941468221, maxMem=4445763010
15/05/31 16:23:00 INFO MemoryStore: Block taskresult_6 stored as bytes in memory (estimated size 383.4 MB, free 2.9 GB)
15/05/31 16:23:00 INFO BlockManagerInfo: Added taskresult_5 in memory on localhost:57022 (size: 383.4 MB, free: 3.6 GB)
15/05/31 16:23:00 INFO BlockManagerMaster: Updated info of block taskresult_5
15/05/31 16:23:00 INFO BlockManagerInfo: Added taskresult_6 in memory on localhost:57022 (size: 383.4 MB, free: 3.3 GB)
15/05/31 16:23:00 INFO BlockManagerMaster: Updated info of block taskresult_6
15/05/31 16:23:01 INFO Executor: Finished task 0.0 in stage 3.0 (TID 5). 401995375 bytes result sent via BlockManager)
15/05/31 16:23:01 INFO Executor: Finished task 1.0 in stage 3.0 (TID 6). 401995375 bytes result sent via BlockManager)
15/05/31 16:28:11 INFO BlockManager: Removing block taskresult_5
15/05/31 16:28:11 INFO MemoryStore: Block taskresult_5 of size 401995375 dropped from memory (free 3504294789)
15/05/31 16:28:11 INFO BlockManagerInfo: Removed taskresult_5 on localhost:57022 in memory (size: 383.4 MB, free: 3.6 GB)
15/05/31 16:28:11 INFO BlockManagerMaster: Updated info of block taskresult_5
15/05/31 16:28:11 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 611140 ms on localhost (1/2)
15/05/31 16:28:12 INFO BlockManager: Removing block taskresult_6
15/05/31 16:28:12 INFO MemoryStore: Block taskresult_6 of size 401995375 dropped from memory (free 3906290164)
15/05/31 16:28:12 INFO BlockManagerInfo: Removed taskresult_6 on localhost:57022 in memory (size: 383.4 MB, free: 4.0 GB)
15/05/31 16:28:12 INFO BlockManagerMaster: Updated info of block taskresult_6
15/05/31 16:28:12 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 6) in 673585 ms on localhost (2/2)
15/05/31 16:28:12 INFO DAGScheduler: Stage 3 (reduce at RDDFunctions.scala:112) finished in 673,993 s
15/05/31 16:28:12 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
15/05/31 16:28:12 INFO DAGScheduler: Job 3 finished: reduce at RDDFunctions.scala:112, took 675,802222 s
15/05/31 16:39:21 INFO BlockManager: Removing broadcast 4
15/05/31 16:39:21 INFO BlockManager: Removing block broadcast_4_piece3
15/05/31 16:39:21 INFO MemoryStore: Block broadcast_4_piece3 of size 4194304 dropped from memory (free 3910484468)
15/05/31 16:39:22 INFO BlockManagerInfo: Removed broadcast_4_piece3 on localhost:57022 in memory (size: 4.0 MB, free: 4.0 GB)
15/05/31 16:39:22 INFO BlockManagerMaster: Updated info of block broadcast_4_piece3
15/05/31 16:39:22 INFO BlockManager: Removing block broadcast_4_piece0
15/05/31 16:39:22 INFO MemoryStore: Block broadcast_4_piece0 of size 4194304 dropped from memory (free 3914678772)
15/05/31 16:39:22 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:57022 in memory (size: 4.0 MB, free: 4.0 GB)
15/05/31 16:39:22 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
15/05/31 16:39:22 INFO BlockManager: Removing block broadcast_4_piece4
15/05/31 16:39:22 INFO MemoryStore: Block broadcast_4_piece4 of size 2169679 dropped from memory (free 3916848451)
15/05/31 16:39:22 INFO BlockManagerInfo: Removed broadcast_4_piece4 on localhost:57022 in memory (size: 2.1 MB, free: 4.0 GB)
15/05/31 16:39:22 INFO BlockManagerMaster: Updated info of block broadcast_4_piece4
15/05/31 16:39:22 INFO BlockManager: Removing block broadcast_4_piece1
15/05/31 16:39:22 INFO MemoryStore: Block broadcast_4_piece1 of size 4194304 dropped from memory (free 3921042755)
15/05/31 16:39:22 INFO BlockManagerInfo: Removed broadcast_4_piece1 on localhost:57022 in memory (size: 4.0 MB, free: 4.0 GB)
15/05/31 16:39:22 INFO BlockManagerMaster: Updated info of block broadcast_4_piece1
15/05/31 16:39:22 INFO BlockManager: Removing block broadcast_4
15/05/31 16:39:22 INFO MemoryStore: Block broadcast_4 of size 400204368 dropped from memory (free 4321247123)
15/05/31 16:39:22 INFO BlockManager: Removing block broadcast_4_piece2
15/05/31 16:39:22 INFO MemoryStore: Block broadcast_4_piece2 of size 4194304 dropped from memory (free 4325441427)
15/05/31 16:39:22 INFO BlockManagerInfo: Removed broadcast_4_piece2 on localhost:57022 in memory (size: 4.0 MB, free: 4.0 GB)
15/05/31 16:39:22 INFO BlockManagerMaster: Updated info of block broadcast_4_piece2
15/05/31 16:39:22 INFO ContextCleaner: Cleaned broadcast 4
java.lang.OutOfMemoryError: Java heap space

samysung
Messages : 2
Inscription : 03 nov. 2011 16:08

Re: spark out of memory error

Message par samysung » 02 juin 2015 13:48

Bonjour,
tu as un problème de mémoire avec la VM java, et non celle de Scala. Le code utilisé doit faire des allocations de mémoires pour des variables java. Tu peux augmenter la mémoire de la JVM utilisée par scala en passant l'option scala JAVA_OPTS="-Xmx512M" (ou scala -j Xmx512M). Là c'est pour 512 MO (tu dois convertir tes Go en Mo). Il y a pas mal d'infos intérressantes sur cette en ce qui concerne le passage de commande ici: http://www.scala-lang.org/files/archive ... scala.html
Cordialement,
Sam.

Crucianu
Messages : 17
Inscription : 21 janv. 2015 18:35

Re: spark out of memory error

Message par Crucianu » 08 juin 2015 10:46

Bonjour,

Différents calculs peuvent utiliser beaucoup de mémoire, les RDD et variables broadcast ne sont pas les seuls consommateurs de mémoire. Faire en sorte que les RDD tiennent en mémoire n'est donc pas suffisant.

Effectivement, une partie de la mémoire de la JVM est utilisée pour la persistance (cache) : spark.storage.memoryFraction = 0.6 par défaut. Si vous n'avez pas besoin d'autant, vous pouvez la réduire. Je vous invite à regarder la page
https://spark.apache.org/docs/latest/tuning.html

Aussi, vous pouvez partitionner plus (2-4 partitions par processeur), cela réduit le volume de mémoire nécessaire.

Bien cordialemant,
Michel Crucianu

Répondre

Qui est en ligne ?

Utilisateurs parcourant ce forum : Aucun utilisateur inscrit et 1 invité