A SHORT HISTORY OF FILE STORAGE AND RELATIONAL DATABASES

Binary file content can be stored directly within a relational database, typically in a Binary Large Object (BLOB) column. While this approach provides the benefit of having both the binary content and its associated relational data managed within the same transactional system, storing large volumes of binary data directly within database tables can introduce significant storage, performance, backup and operational overhead. An alternative is to store binary file content outside the database. However, with this approach the database loses its inherent ability to guarantee transactional consistency between the relational data and the external file content. Responsibility for maintaining consistency between the two storage systems must then be assumed by the application layer.

In 2007 and 2008 Oracle and Microsoft introduced specialized technologies to address aspects of the problem described above: SecureFiles and FILESTREAM respectively. These technologies provide more sophisticated mechanisms for managing large binary objects while retaining important database integration capabilities. In particular, SQL Server FILESTREAM allows binary file content to be stored in the filesystem while maintaining transactional consistency with the associated relational database records.

However, database-integrated file storage technologies such as FILESTREAM are not designed to address every requirement of extremely large, geographically distributed storage architectures. For requirements involving petabyte-scale volumes, massive horizontal scalability and geographically distributed access, cloud object storage services such as Microsoft Azure Blob Storage, Amazon S3 and Google Cloud Storage offer significant advantages. These services provide highly scalable, centrally managed and cost-effective storage infrastructure capable of accommodating enormous volumes of data.

The architectural trade-off of cloud object storage is that it operates independently of the relational database. While object storage services provide their own durability, consistency and access-control guarantees, they do not ordinarily participate in the same ACID transaction as an independently managed relational database. Consequently, transactional consistency between document content and associated relational data must be maintained through application-layer coordination, additional infrastructure or compensating transaction mechanisms.

Cloud object storage also introduces a separate security boundary. Access to document content is governed by the object storage service's own identity and access-control mechanisms, in addition to those governing the relational database. This introduces additional complexity when the objective is to have the database independently establish the originating user's identity and enforce access permissions over both document content and its associated relational data. These challenges are manageable, and cloud-based architectures can provide very high levels of reliability and security. However they require compromises in the nature of the integrity and security guarantees that can be provided. In particular, they do not natively provide an integrated architecture in which the relational database retains transactional control of the binary document content while independently establishing the originating user's identity and enforcing database-controlled access permissions.

The above distinction is fundamental to Elyse. By retaining full database control of document binary content through SQL Server FILESTREAM, and combining this with Windows Integrated Security and database-enforced, ORIGINAL_LOGIN()-based identity and access controls, Elyse provides a combination of transactional integrity and originating-user security guarantees that is not presently available as an integrated capability in conventional cloud-native managed database and object-storage services. Although such an architecture could theoretically be engineered in a future cloud platform, the necessary combination of capabilities is not presently offered as a native managed cloud service.