月编程做网站,wordpress 指定目录页,千锋教育地址,企业关键词优化专业公司Spark RDD分区2G限制 文章目录Spark RDD分区2G限制问题现象解决方法为什么2G限制个人思#xff08;yu#xff09;考#xff08;jian#xff09;问题现象
遇到这个问题时#xff0c;spark日志会报如下的日志
片段1#xff1a;
15/04/16 14:13:03 WARN scheduler.TaskSe…Spark RDD分区2G限制 文章目录Spark RDD分区2G限制问题现象解决方法为什么2G限制个人思yu考jian问题现象
遇到这个问题时spark日志会报如下的日志
片段1
15/04/16 14:13:03 WARN scheduler.TaskSetManager: Lost task 19.0 in stage 6.0 (TID 120, 10.215.149.47): java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:828)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:123)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:132)
at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:517)
at org.apache.spark.storage.BlockManager.getLocal(BlockManager.scala:432)
at org.apache.spark.storage.BlockManager.get(BlockManager.scala:618)
at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:146)
at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)片段2
15/04/16 14:19:45 INFO scheduler.TaskSetManager: Starting task 20.2 in stage 6.0 (TID 146, 10.196.151.213, PROCESS_LOCAL, 1666 bytes)15/04/16 14:19:45 INFO scheduler.TaskSetManager: Lost task 20.2 in stage 6.0 (TID 146) on executor 10.196.151.213: java.lang.IllegalArgumentException (Size exceeds Integer.MAX_VALUE) [duplicate 1]15/04/16 14:19:45 INFO scheduler.TaskSetManager: Starting task 20.3 in stage 6.0 (TID 147, 10.196.151.213, PROCESS_LOCAL, 1666 bytes)15/04/16 14:19:45 INFO scheduler.TaskSetManager: Lost task 20.3 in stage 6.0 (TID 147) on executor 10.196.151.213: java.lang.IllegalArgumentException (Size exceeds Integer.MAX_VALUE) [duplicate 2]15/04/16 14:19:45 ERROR scheduler.TaskSetManager: Task 20 in stage 6.0 failed 4 times; aborting job15/04/16 14:19:45 INFO cluster.YarnClusterScheduler: Cancelling stage 615/04/16 14:19:45 INFO cluster.YarnClusterScheduler: Stage 6 was cancelled15/04/16 14:19:45 INFO scheduler.DAGScheduler: Job 6 failed: collectAsMap at DecisionTree.scala:653, took 239.760845 s15/04/16 14:19:45 ERROR yarn.ApplicationMaster: User class threw exception: Job aborted due to stage failure: Task 20 in stage 6.0 failed 4 times, most recent failure: Lost task 20.3 in stage 6.0 (TID 147, 10.196.151.213): java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUEat sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:828)异常就是某个partition的数据量超过了Integer.MAX_VALUE2147483647 2GB
解决方法
手动设置RDD的分区数量。当前使用的Spark默认RDD分区是18个后来手动设置为1000个上面这个问题就迎刃而解了。可以在RDD加载后使用RDD.repartition(numPart:Int)函数重新设置分区数量。
为什么2G限制
目前spark社区对这个限制有很多讨tu论caospark官方团队已经注意到了这个问题但是直到1.2版本这个问题还是没有解决。因为牵涉到整个RDD的实现框架所以改进成本相当大!
下面是一些相关的资料有兴趣的读者可以进一步的阅读:
2GB limit in spark for blockscreate LargeByteBuffer abstraction for eliminating 2GB limit on blocksWhy does Spark RDD partition has 2GB limit for HDFS抛异常的java代码:FileChannelImpl.java
个人思yu考jian
这个限制有一定合理性。因为RDD中partition的操作是并发执行的如果partition量过少导致并发数过少会限制计算效率。所以基于这个限制spark应用程序开发者会主动扩大partition数量也就是加大并发量最终提高计算性能。
转载自https://www.cnblogs.com/bourneli/p/4456109.html
