How do I suppress the bloat of useless information when using the DUMP command while using grunt via 'pig -x local'?

StackOverflow https://stackoverflow.com/questions/16410489

Question

I'm working with PigLatin, using grunt, and every time I 'dump' stuffs, my console gets clobbered with blah blah, blah non-info, is there a way to surpress all that? 

grunt> A = LOAD 'testingData' USING PigStorage(':'); dump A;

2013-05-06 19:42:04,146 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-05-06 19:42:04,147 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false ...
...
--- another like 50 lines of useless context clobbering junk here... till --- ...
...
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

now my like 4 lines of info looking for: 

(daemon,*,1,1,System Services,/var/root,/usr/bin/false)
(uucp,*,,,/var/spool/uucp,/usr/sbin/uucico)
(taskgated,*,13,13,Task Gate Daemon,/var/empty,/usr/bin/false)
(networkd,*,24,24,Network Services,/var/empty,/usr/bin/false)
(installassistant,*,25,25,/usr/bin/false)

grunt>

---> obviously if it errors, fine lotsa info helpful, but not when it basically works great.

Was it helpful?

Solution

You need to set the log4j properties. For example:

$PIG_HOME/conf/pig.properties :
enable:
# log4jconf=./conf/log4j.properties

rename: log4j.properties.template -> log4j.properties

log4j.properties :
set info to error:
log4j.logger.org.apache.pig=info, A

You may also set the Hadoop related logging level as well: 

log4j.logger.org.apache.hadoop = error, A

OTHER TIPS

An easy way to do this seems to be to redirect standard error as below.

But it will suppress all errors.

pig -x local 2> /dev/null

Also found that if you remove or rename your hadoop install directory to basically make it inaccessible to pig then all those INFO messages go away.
Changing logging levels in hadoop didn't help, just so that you know.

When you start pig, pass it a log4j.properties file with pig -4 <filename>.

In my case there was a log4j.properties in the conf directory and setting the level of the logger named org.apache.pig to ERROR is sufficient to make the logger less verbose.

log4j.logger.org.apache.pig=ERROR, A

pig has debug log level one need to set that in pig.properties file, 

# Logging level. debug=OFF|ERROR|WARN|INFO|DEBUG (default: INFO)
#
# debug=INFO

The reason one get large logs on console, e.g. change it to ERROR

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top