Having a schema is part of the definition of a relational database, so a database can not be both schemaless and relational.
But when you are searching for a database which is good for modeling and analyzing relations between entities without enforcing a consistent schema, you could take a look at graph databases like Neo4j. These databases focus on defining entities mostly by their relation to other entities. They make it really easy to find entities which have common relations to other entities.