什么是Spark中的RDD依赖项？(What is RDD dependency in Spark?)

什么是Spark中的RDD依赖项？(What is RDD dependency in Spark?)

据我所知有两种类型的依赖关系： 狭窄和广泛 。 但我不明白依赖关系如何影响子RDD 。 是仅含子RDD的元数据，其中包含如何从父RDD构建新的RDD块的信息？ 还是子RDD是由父RDD创建的自给自足的数据集？

As I know there are two types of dependencies: narrow & wide. But I dont understand how dependency affects to child RDD. Is child RDD only metadata which contains info how to build new RDD blocks from parent RDD? Or child RDD is self-sufficient set of data which was created from parent RDD?

最满意答案

是的，子RDD是描述如何从父RDD计算RDD的元数据。

考虑一下org/apache/spark/rdd/MappedRDD.scala例如：

private[spark] class MappedRDD[U: ClassTag, T: ClassTag](prev: RDD[T], f: T => U) extends RDD[U](prev) { override def getPartiti: Array[Partition] = firstParent[T].partiti override def compute(split: Partition, context: TaskContext) = firstParent[T].iterator(split, context).map(f) }

当你说rdd2 = (...) ， rdd2将会是这样一个MappedRDD 。 compute仅在稍后执行，例如，当您调用 。

即使RDD没有父母（例如(...) ），RDD也始终是这样的元数据。 RDD存储在节点上的唯一情况是，如果您将其标记为使用进行缓存，然后导致它被计算。

另一个类似的情况是调用 。 此功能标记检查点的RDD。 下一次计算时，它将被写入磁盘，随后访问RDD将导致它从磁盘读取而不是重新计算。

cache和checkpoint之间的区别在于缓存的RDD仍然保留其依赖关系。 高速缓存的数据可能在内存压力下丢弃，可能需要部分或全部重新计算。 这在检查点RDD中不会发生，所以依赖关系在那里被丢弃。

Yes, the child RDD is metadata that describes how to calculate the RDD from the parent RDD.

Cider org/apache/spark/rdd/MappedRDD.scala for example:

private[spark] class MappedRDD[U: ClassTag, T: ClassTag](prev: RDD[T], f: T => U) extends RDD[U](prev) { override def getPartiti: Array[Partition] = firstParent[T].partiti override def compute(split: Partition, context: TaskContext) = firstParent[T].iterator(split, context).map(f) }

When you say rdd2 = (...), rdd2 will be such a MappedRDD. compute is only executed later, for example when you call .

An RDD is always such a metadata, even if it has no parents (for example (...)). The only case an RDD is stored on the nodes, is if you mark it for caching with , and then cause it to be computed.

Another similar situation is calling . This function marks the RDD for checkpointing. The next time it is computed, it will be written to disk, and later access to the RDD will cause it to be read from disk instead of recalculated.

The difference between cache and checkpoint is that a cached RDD still retains its dependencies. The cached data can be discarded under memory pressure, and may need to be recalculated in part or whole. This cannot happen with a checkpointed RDD, so the dependencies are discarded there.