To check for monotonicity, remember the last value and require the current value to be greater. Taking your code as template:
xmlNodePtr node;
double last_position = -1;
for (node = renderer-root->children; node; node = node->next) {
if ((XML_ELEMENT_NODE == node->type) && xmlStrEqual((const xmlChar *)"point", node->name)) {
xmlChar *position_str = xmlGetProp(node, (const xmlChar *)"position");
double position = strtod(position_str, NULL);
xmlFree(position_str);
if (position <= last_position) {
/* position out of order */
return;
}
...