为什么Spark中的示例代码不会加载到spark中
下面的代码示例来自Advanced Analytics with Spark一书。 当我将它加载到spark-shell（版本1.4.1）时，它会给出以下错误，指示它找不到StatCounter：
import org.apache.spark.util.StatCounter
<console>:9: error: not found: type StatCounter
val stats: StatCounter = new StatCounter()
^
<console>:9: error: not found: type StatCounter
val stats: StatCounter = new StatCounter()
^
<console>:23: error: not found: type NAStatCounter
def apply(x: Double) = new NAStatCounter().add(x)
如果我只是在spark-shell中执行以下操作，则不存在任何问题：
scala> import org.apache.spark.util.StatCounter
import org.apache.spark.util.StatCounter
scala> val statsCounter: StatCounter = new StatCounter()
statsCounter: org.apache.spark.util.StatCounter = (count: 0, mean: 0.000000, stdev: NaN, max: -Infinity, min: Infinity)
问题似乎是在spark-shell中使用：load命令。
代码如下：
import org.apache.spark.util.StatCounter
class NAStatCounter extends Serializable {
val stats: StatCounter = new StatCounter()
var missing: Long = 0
def add(x: Double): NAStatCounter = {
if (java.lang.Double.isNaN(x)) {
missing += 1
} else {
stats.merge(x)
}
this
}
def merge(other: NAStatCounter): NAStatCounter = {
stats.merge(other.stats)
missing += other.missing
this
}
override def toString = {
"stats: " + stats.toString + " NaN: " + missing
}
}
object NAStatCounter extends Serializable {
def apply(x: Double) = new NAStatCounter().add(x)
}
我和你有完全相同的问题。
我试着解决它，
更改
val stats: StatCounter = new StatCounter()
INTO
val stats: org.apache.spark.util.StatCounter = new org.apache.spark.util.StatCounter()
原因可能是系统不知道StatCounter的路径
链接地址: http://www.djcxy.com/p/31623.html