I'm not sure if I understand the question correctly, but if you want to use the offset for all posts (not just the ones that are tagged with term 18 or 23), I'm thinking you could use a sub query:
// the sub query will select a maximum of 50 posts starting from the supplied $offset
// notice that it will retrieve max 50 posts disregarding associated tags
// you may want to move the 'ORDER BY' clause into the sub query as well
$query = "SELECT p.* FROM (
SELECT * FROM wp_posts
ORDER BY post_date
WHERE post_type = 'post' && post_status = 'publish'
LIMIT $offset, 50
) p
INNER JOIN wp_term_relationships tr ON tr.object_id = p.id && tr.taxonomy_term_id IN (23, 18)
INNER JOIN wp_terms t ON t.term_id = tr.taxonomy_term_id
ORDER BY p.post_date DESC";
$the_query = new WP_Query($query);