starting mongodb via numactl as daemon
Domanda
I'm trying to get mongodb started on a NUMA machine as a daemon. When I run
numactl --interleave=all mongod &
Mongo starts and runs correctly, but all the output still shows up. (e.g., Fri Jun 22 12:10:29 [initandlisten] connection accepted from 127.0.1.1:51837)
However, when I start mongo on its own (like below), it fails (logs below):
service mongodb start
I get the following in the logs
Fri Jun 22 12:08:41 [initandlisten] MongoDB starting : pid=3348 port=27017 dbpath=/var/lib/mongodb 64-bit host=beckett
Fri Jun 22 12:08:41 [initandlisten]
Fri Jun 22 12:08:41 [initandlisten] ** WARNING: You are running on a NUMA machine.
Fri Jun 22 12:08:41 [initandlisten] ** We suggest launching mongod like this to avoid performance problems:
Fri Jun 22 12:08:41 [initandlisten] ** numactl --interleave=all mongod [other options]
Fri Jun 22 12:08:41 [initandlisten]
Fri Jun 22 12:08:41 [initandlisten] db version v2.0.6, pdfile version 4.5
Fri Jun 22 12:08:41 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Fri Jun 22 12:08:41 [initandlisten] build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Fri Jun 22 12:08:41 [initandlisten] options: { auth: "true", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Fri Jun 22 12:08:41 [initandlisten] journal dir=/var/lib/mongodb/journal
Fri Jun 22 12:08:41 [initandlisten] recover : no journal files present, no recovery needed
Fri Jun 22 12:08:42 [initandlisten] couldn't open /var/lib/mongodb/admin.ns errno:13 Permission denied
Fri Jun 22 12:08:42 [initandlisten] error couldn't open file /var/lib/mongodb/admin.ns terminating
Fri Jun 22 12:08:42 dbexit:
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to close listening sockets...
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to flush diaglog...
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to close sockets...
Fri Jun 22 12:08:42 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jun 22 12:08:42 [initandlisten] shutdown: lock for final commit...
Fri Jun 22 12:08:42 [initandlisten] shutdown: final commit...
Fri Jun 22 12:08:42 [initandlisten] shutdown: closing all files...
Fri Jun 22 12:08:42 [initandlisten] closeAllFiles() finished
Fri Jun 22 12:08:42 [initandlisten] journalCleanup...
Fri Jun 22 12:08:42 [initandlisten] removeJournalFiles
Fri Jun 22 12:08:42 [initandlisten] shutdown: removing fs lock...
Fri Jun 22 12:08:42 dbexit: really exiting now
I don't know how admin.ns could have a permissions issue while I'm running as root or why when wrapped in numactl it starts up ok. Ideally, I'd like to use numactl in the start_server() function like so:
start_server(){
/usr/bin/numactl --interleave=all -- \
start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
--make-pidfile --chuid $DAEMONUSER \
--exec $DAEMON -- $DAEMON_OPTS
errcode=$?
return $errcode
}
Bottom line, how can I get mongo to start as a daemon on a NUMA machine?
Soluzione
Turns out my problem was a combination of numa and permissions issues. Thanks, @Mark for the help. To start mongodb as a daemon on a NUMA setup, replace the start_server() function in /etc/init.d/mongodb with the following:
start_server() {
# check for numactl
NUMACTL=$(which numactl)
if [ -n "$NUMACTL" ]; then
DAEMON_OPTS="--interleave=all ${DAEMON} ${DAEMON_OPTS}"
DAEMON="$NUMACTL"
fi
# Start the process using the wrapper
/usr/bin/numactl --interleave=all -- \
start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
--make-pidfile --chuid $DAEMONUSER \
--exec $DAEMON -- $DAEMON_OPTS
errcode=$?
return $errcode
}
Altri suggerimenti
I presume you know the usual warnings (http://www.mongodb.org/display/DOCS/NUMA) about 'mongo & numa' so I won't go on about them.
Here's a sample upstart configuration file for mongodb with numa - https://gist.github.com/1364716.
Based on this Google Group thread, the following lines were added to the start_server function in the init script and it was successful -
start_server() {
# check for numactl
NUMACTL=$(which numactl)
if [ -n "$NUMACTL" ]; then
DAEMON_OPTS="--interleave=all ${DAEMON} ${DAEMON_OPTS}"
DAEMON="$NUMACTL"
fi
# Start the process using the wrapper
for ubuntu 16.04
[Unit]
Description=High-performance, schema-free document-oriented database
After=time-sync.target network.target
[Service]
Type=forking
User=mongod
Group=mongod
LimitNOFILE=65000
PermissionsStartOnly=true
EnvironmentFile=/etc/default/mongod
ExecStartPre=/usr/bin/percona-server-mongodb-helper.sh
ExecStart=/usr/bin/env bash -c "numactl --interleave=all /usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
#ExecStart=/usr/bin/env bash -c "/usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
PIDFile=/var/run/mongod.pid
[Install]
WantedBy=multi-user.target
the two solutions above did no work for me, so here follows what got my mongodb running:
replace the start_server()
function in /etc/init.d/mongodb
for the code below
start_server() {
test -e "$RUNDIR" || install -m 755 -o mongodb -g mongodb -d "$RUNDIR"
NUMACTL=$(which numactl)
if [ ! "$NUMACTL" ]; then
# start original
start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
errcode=$?
return $errcode
else
# Start the process using the wrapper
$NUMACTL --interleave=all -- start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
errcode=$?
return $errcode
fi
}
for ubuntu 14
# apt-get -y install numactl
add to /etc/sysctl.conf line vm.zone_reclaim_mode = 0
# sysctl -p
# service mongod restart