MBAS Heterogenous Storage – Atlanta – June 2019
Graham “show-me-the-code” Tomkins
Introduction to Storage Cake: (Cake Level: Jaffa Cake)
Currently one of the ways to store actual files in CDS (CRM/XrM/XRM/Xrm/CrM/Dynamics/CE/D365) is to utilise and force notes with attachments into the GUI – relatively ugly and non-jaffa-cake-like. This was do-able easily but the storage of attachments, either out of the back of processes i.e. azure logs, contract storage, image audio etc. was normally fed in through code.
Accessing these files proved simple but frustrating, also the one image field per entity was circumvented with, sometimes unnecessary child entities – un-normalisation is no-ones friend! There was also a size and resolution limit that restricted many fundamental uses for the image field.
New Cake: (Cake Level: Profiterole)
During the MBAS session “Heterogeneous Storage…”<link> – the speakers lifted the lid on what is coming – this article, with all it’s sarcasm is my summary of the session, which covered:
- Data Types
- Auditing & Logs
- New File Types
Licensing: (Cake Level: Petit Four)
Licensing <link> around Microsoft products has always been a dirty word, I find understanding quantum mechanics easier than attaining the PhD required to give a simple clear answer to most MS CRM/CDS/XRM/Dynamics/PowerApps/CE based licensing questions…
The storage aspect has got slightly more complex, however I believe this is a good thing as the previous one size fits all approach isn’t good for todays usage of the platform. The PowerPoint gives more detail on the PowerApps storage, however the slide below relates specifically to CRM (D365/CE/XRM etc.etc.)
Now I believe the clients will have needed to switch to the new licensing model (automatic at the next renewal period start) in order for the above to apply. As you can see the database size (old money: MDF file size) is now split into Data (flour and eggs), File Data (butter) and Logs (sugar).
This will most likely help remove laziness on the system deployers front where logs are cleared infrequently if ever. It also looks like they are treating all BASE64 blobs in SQL very differently (why so much cheaper I wonder….)
Data Types: (Cake Level: Rocky Road and Custard)
From the slide extract below it’s clear that they are also going to treat indexes and search metadata differently also. In my experience, separating this out at the SQL level can yield performance improvements – I assume this already happens in the mystical world of PowerPlatform/CRM/D365/CE/CDS/XRM SQL 😊
Relational Data is the basis of all the good stuff inside CRM – it’s most field types stored in entities and tables within the system – from the system tables all the way up to the custom tables you added to store information about your pet’s food preferences.
Files, Images, Attachments and Other Binaries is, it seems, anything we used to store as BASE64 data types in SQL, it’s your files serialised into SQL taking up crazy amounts of memory and disk space for infrequent access, all the email attachments, and will be used for the 2 new file types (see below).
Audit data etc. is the small amount of space (that never grows automatically…) we will be given to store standard system audit logs, plug-in trace logs (oh we need to make sure this is all turned off in dev…) etc. Now I believe this is different in space to the Office 365 access logs however I’m not entirely sure.
N.B: it was alluded to, that coming data types would include observational IOT data i.e. events from devices… inter-ma-resting.
Auditing & Logs: (Cake Level: Peanut Butter Cheesecake with Cream)
With the new restrictions on size, MS are going to be giving us new tooling to help clean up the Audit logs in different ways (other than the one size fits all by date that currently exists).
By Entity will help us clean up when the PetFoodPlugin.dll has gone crazy and sequentially update all dogs to liking fish instead of chicken.
Access logs by people and systems will help correct for when my specific personal workflow has been executed across 9 million pet hamster records.
All logs by date is the good old sweep up.
New Field Types: (Cake Level: Black Forrest Gateau)
Ok so storing both images and files against current CRM records is a pain in the yaris. Two new data types for fields (attributes if you must) are on their way
- Multiple Files allowed per record
- Configurable Size limits per field
- Upto 128mb
- Binary Access
- Searching Inside files is coming later
- Full High Resolution Images Support
- Thumbnail Generation on Upload
- Any entity
- Backwards Compatible
- N.B: It was unclear to me, even with a question, if we are going to be allowed to store multiple image fields per entity – I’m going to assume that we will be allowed (as it’s currently limited to one field per entity).
All in all, a very informative session – the team were honest about the timelines being the end of 2019 (ish) – which gives us enough time to make sure any future looking designs can incorporate and make the most the new fields. With smart moves shuffling around data types behind the scenes, you can see they are aiming to turn on the new version of the old SharePoint File search – but I assume done properly and with a correctly controlled scale. And to finish – some cake: