Can PostGIS or Oracle Spatial and Graph scale to hundreds of terabytes of data, or should I use a NoSQL option like DynamoDB? [closed]

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Database Administrators Stack Exchange.

Closed 1 year ago.

Improve this question

I'm going to have a very large data set (a few hundred terabytes big eventually), but it is very well structured, relatively simple data involving lat, long points (which is why I want some GIS compatibility).

From what I can tell PostGIS can only handle 32TB or so, and I'm unsure what Oracle Spatial can scale to. Amazon's DynamoDB can go up to petabyte scale, but I've read complaints about how it can become rather complicated and be avoided without having a good reason to use it. And using NoSQL for such structured data seems wrong, but I can't find other alternatives for this size. As far as costs go, yeah it's going to be expensive but let's assume that's not a huge problem. Retrieval speed for spatial and temporal queries is the main deciding factor.

Answer 1

You might want to consider doing some Requirements Engineering for your project. When you have all the requirements listed, then you will want to write it up in a document and have companies like Oracle, Microsoft, IBM or others offer you a solution.

I don't think an open-source solution will work for the amount of data you are expecting without having adequate consultants in the background. The exception to the rule being that you are willing to go through some painful "learning-by-doing" experiences. This can be a viable solution if your "customers" (internal or external) are willing to live with some down-time during the various growth phases of the project.

I guess any database could potentially scale to some amount of data depending on the framework you would have to build around it to meet your requirements.

There are cases of large database systems which are outlined in the following documents/links. Some of the databases are reduced in size based on features the RDBMS supply:

Microsoft SQL Server

• NASDAQ OMX Group Reduces 2 PB of Data to 500 TB with Microsoft In-Memory Technology (Microsoft Customers)
• Microsoft Applauds 1.1 Petabytes SQL Server 2008 Database (Softpedia)
• Updated database size survey results (Paul Randal's SQL Skills)

PostgreSQL

• Can a PostgreSQL database instance support petabytes of data? (Quora)
• Updating a 50 terabyte PostgreSQL database (medium.com)
• What is the maximum size for a row, a table, and a database? (PostgreSQL Wiki)

  Of course, these are not actually unlimited, but limited to available disk space and memory/swap space. Performance may suffer when these values get unusually large.

PostGIS

• Introduction to PostGIS (PostGIS)

  GlobeXplorer is a web-based service providing online access to petabytes of global satellite and aerial imagery. GlobeXplorer uses PostGIS to manage the metadata associated with the imagery catalogue, so queries for imagery first search the PostGIS catalogue to find the location of the relevant images, then pull the images from storage and return them to the client. In building their system, GlobeXplorer tried other spatial databases but eventually settled on PostGIS because of the great combination of price and performance it offers.

Oracle Spatial and Graph

• Unstructured Data Management with Oracle Database 12c (Oracle White Paper PDF)

  The size limit for individual media objects that can be stored and retrieved within database storage structures (BLOB) is 128 terabytes.

So basically the limitations are probably to be found more within the hardware you have available than within the software used. But that depends.