Really low accuracy with iris dataset

adwivedi · May 17, 2019, 7:15am

Can someone help me figure out how to get the correct predicted class from the predicted probabilities.
Right now the predicted class labels aren’t the same as the index of the argmax which results in incorrect calculation of accuracy.

Dataset: http://dpaste.com/2BWHTZ8
Here’s my program using the scala API

import org.apache.mxnet._
import org.apache.mxnet.io.NDArrayIter
import org.apache.mxnet.optimizer.SGD

object ScalaApiExample {

  val DATA_PATH = "/home/a/iris.csv"

  def getInput(): DataIter = {
    val lines = scala.io.Source.fromFile(DATA_PATH).mkString.split("\n")

    val array = (lines:Array[String], f:Seq[Double] => Seq[Double]) => lines.map({ line =>
      val Array(_, row @ _*) = line.split(",")
      f(row.map(_.toDouble))
    })

    val targets = NDArray.array(array(lines, line => Array(line.last)).flatMap(identity), Shape(150))
    val feats = NDArray.array(array(lines, line => line.init).flatMap(identity), Shape(150, 4))

    new NDArrayIter.Builder()
      .addData("data", feats)
      .addLabel("softmax_label", label = targets)
      .setBatchSize(128).build()
  }


  def accuracy(preds: NDArray, labels: NDArray) = {
    val sum = (preds.toArray zip labels.toArray).map({
      case(y, yPred) =>
        if(y == yPred) 1 else 0
    }).sum
    sum.toFloat / preds.size
  }



  def getNetwork = {
    val data = Symbol.Variable("data")
    val fc1 = Symbol.FullyConnected(name = "fc1")()(Map("data" -> data, "num_hidden" -> 128))
    val act1 = Symbol.Activation(name = "relu1")()(Map("data" -> fc1, "act_type" -> "relu"))
    val fc2 = Symbol.FullyConnected(name = "fc2")()(Map("data" -> act1, "num_hidden" -> 128))
    val act2 = Symbol.Activation(name = "relu2")()(Map("data" -> fc2, "act_type" -> "relu"))
    val fc3 = Symbol.FullyConnected(name = "fc3")()(Map("data" -> act2, "num_hidden" -> 3))
    val mlp = Symbol.SoftmaxOutput(name = "softmax")()(Map("data" -> fc3))
    mlp
  }

  def main(args: Array[String]): Unit = {
    val dataIter = getInput

    val model = FeedForward.newBuilder(getNetwork)
      .setContext(Context.cpu())
      .setNumEpoch(2000)
      .setOptimizer(new SGD(learningRate = 0.01f, momentum = 0.9f, wd = 0.0001f))
      .setTrainData(dataIter)
      .setEvalData(dataIter)
      .build()

    model.fit(dataIter,
      dataIter,
      new Accuracy(),
      KVStore.create("local")
    )

    val probs = model.predict(dataIter)
    val preds = NDArray.argmax_channel(probs(0))
    val acc = accuracy(preds, dataIter.getLabel().head)

    println(s"accuracy = ${acc}")
  }
}

mouryarishik · May 17, 2019, 2:24pm

Though I don’t use scala, so I can’t say if anything wrong with the code. But I can give a suggestion, make sure you have normalized the data.
You can do any one of the following:

Converting all values of dataset to range 0 and 1. You can achieve that as follows:
dataset = (dataset - min)/(max - min)
where min is the minimum value of all feature columns.
Normalising using mean and standard deviation of your feature columns. You can achieve that as follows:
dataset = (dataset - mean)/std
where mean and std are the mean and standard deviation of feature columns.

Sergey · May 17, 2019, 9:37pm

So, are you trying to say that despite having high accuracy, the output is actually wrong? Because, if I run your example, with fixing conversion from Double to Float then I get the following output (last few epochs):

2019-05-17 14:34:32,708 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1997] Train-accuracy=0.98828125
2019-05-17 14:34:32,708 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1997] Time cost=3
2019-05-17 14:34:32,709 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1997] Validation-accuracy=0.98828125
2019-05-17 14:34:32,713 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1998] Train-accuracy=0.98828125
2019-05-17 14:34:32,713 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1998] Time cost=4
2019-05-17 14:34:32,714 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1998] Validation-accuracy=0.98828125
2019-05-17 14:34:32,717 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1999] Train-accuracy=0.98828125
2019-05-17 14:34:32,717 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1999] Time cost=3
2019-05-17 14:34:32,718 [main] [org.apache.mxnet.Model] [INFO] - Epoch[1999] Validation-accuracy=0.98828125

Validation accuracy of 0.988 is really high.

adwivedi · May 22, 2019, 6:10am

Thanks for the suggestion @mourarishik but I am already using the normalized dataset.

adwivedi · May 22, 2019, 6:23am

@Sergey Yes you are write, my bad. The problem was that I was using dataIter.getLabel().head to get the labels, in the calculation for the accuracy, which only gives it for a batch.

However can you tell me which conversion did you fix?

Sergey · May 22, 2019, 5:53pm

In my case, I had to convert numbers from Double to Float. Otherwise, it didn’t want to compile it. Here is my full runnable code, which produced me the accuracy I was posting before (it is 99% same as yours):

package mxnet

import org.apache.mxnet._
import org.apache.mxnet.io.NDArrayIter
import org.apache.mxnet.optimizer.SGD

/**
 * @author ${user.name}
 */
object App {

  val DATA_PATH = "/home/a/iris.csv"

  def getInput(): DataIter = {
    val lines = scala.io.Source.fromFile(DATA_PATH).mkString.split("\n")

    val array = (lines:Array[String], f:Seq[Float] => Seq[Float]) => lines.map({ line =>
      val Array(_, row @ _*) = line.split(",")
      f(row.map(_.toFloat))
    })

    val targets = NDArray.array(array(lines, line => Array(line.last)).flatMap(identity), Shape(150))
    val feats = NDArray.array(array(lines, line => line.init).flatMap(identity), Shape(150, 4))

    new NDArrayIter.Builder()
      .addData("data", feats)
      .addLabel("softmax_label", label = targets)
      .setBatchSize(128).build()
  }


  def accuracy(preds: NDArray, labels: NDArray) = {
    val sum = (preds.toArray zip labels.toArray).map({
      case(y, yPred) =>
        if(y == yPred) 1 else 0
    }).sum
    sum.toFloat / preds.size
  }



  def getNetwork = {
    val data = Symbol.Variable("data")
    val fc1 = Symbol.FullyConnected(name = "fc1")()(Map("data" -> data, "num_hidden" -> 128))
    val act1 = Symbol.Activation(name = "relu1")()(Map("data" -> fc1, "act_type" -> "relu"))
    val fc2 = Symbol.FullyConnected(name = "fc2")()(Map("data" -> act1, "num_hidden" -> 128))
    val act2 = Symbol.Activation(name = "relu2")()(Map("data" -> fc2, "act_type" -> "relu"))
    val fc3 = Symbol.FullyConnected(name = "fc3")()(Map("data" -> act2, "num_hidden" -> 3))
    val mlp = Symbol.SoftmaxOutput(name = "softmax")()(Map("data" -> fc3))
    mlp
  }

  def main(args: Array[String]): Unit = {
    val dataIter = getInput

    val model = FeedForward.newBuilder(getNetwork)
      .setContext(Context.cpu())
      .setNumEpoch(2000)
      .setOptimizer(new SGD(learningRate = 0.01f, momentum = 0.9f, wd = 0.0001f))
      .setTrainData(dataIter)
      .setEvalData(dataIter)
      .build()

    model.fit(dataIter,
      dataIter,
      new Accuracy(),
      KVStore.create("local")
    )

    val probs = model.predict(dataIter)
    val preds = NDArray.argmax_channel(probs(0))
    val acc = accuracy(preds, dataIter.getLabel().head)

    println(s"accuracy = ${acc}")
  }
}

Topic		Replies	Views
How to get individual class accuracy in image-classification on validation set Discussion	1	1116	May 9, 2018
How can I compute the accuracy for a multi-lable dataset? Discussion	1	450	April 25, 2019
Terrible classification accuracy of mxnet	2	874	November 15, 2017
Image classification example accuracy issue Discussion	3	580	September 9, 2019
Biased prediction in image recognition in R 3.5.0 Performance	3	647	June 18, 2018

Really low accuracy with iris dataset

Related Topics