Error when setting mapred.map.tasks in pseudo-distributed mode

Question

As suggested here, I am running hadoop in pseudodistributed mode with the following mapred-site.xml file. The job is running on a 4 core machine.

<configuration>
   <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>  
   </property>
   <property>
     <name>mapred.map.tasks</name> 
     <value>4</value> 
  </property>
  <property>
     <name>mapred.reduce.tasks</name> 
     <value>4</value> 
  </property>

</configuration>

I am getting the following error:

The ratio of reported blocks 1.0000 has reached the threshold 0.9990.

What is the reason for this. How to fix this?

Answer 1

It's not a problem, but a normal procedure during hadoop startup.

Safemode is an HDFS state in which the file system is mounted read-only; no replication is performed, nor can files be created or deleted. This is automatically entered as the NameNode starts, to allow all DataNodes time to check in with the NameNode and announce which blocks they hold, before the NameNode determines which blocks are under-replicated, etc.

You can also see what is Safemode in this page.

The NameNode waits until a specific percentage of the blocks are present and accounted-for, the percentage is set by dfs.safemode.threshold.pct parameter in hdfs-default.xml by default. But you can change it in hdfs-site.xml.

It's

<property>
  <name>dfs.safemode.threshold.pct</name>
  <value>0.999f</value>
  <description>
    Specifies the percentage of blocks that should satisfy 
    the minimal replication requirement defined by dfs.replication.min.
    Values less than or equal to 0 mean not to wait for any particular
    percentage of blocks before exiting safemode.
    Values greater than 1 will make safe mode permanent.
  </description>
</property>

So you see the log ends with 0.9990

The ratio of reported blocks 1.0000 has reached the threshold 0.9990.