Technical insights about a new digital assets management tool: Dammi

In this blogpost I’ll go into the technical details of Dammi, the tool we built to easily locate, store, manage, embed and share images.

Dammi is a digital assets management system that is comprised of a backend API and a frontend JS consumer. Users can upload their PDFs, PSDs, JPEGs and other assets by various means and enrich them with a plethora of metadata. The latter of which will facilitate asset retrieval with the help of elasticsearchAsset metadata is currently restricted to user input (i.e.: tags, hierarchical categories, titles, …) but will be expanded upon with more file-centric data (e.g.: image Exif data) as to decrease this boresome hurdle.

manage-dammi.io_
Dammi allows the user to easily store and manage thousands of images.

Dammi workflow: upload and multi edit

Users can upload their files using the default html upload field, drag ‘n drop or by mailing them to their designated dammi.io email address. We’re planning on adding more ways to get your files into Dammi (e.g.: Dropbox, ftp, …)

Once the files have been uploaded to our API (Symfony) the files can be enriched with metadata using our single or multi file edit interfaces.

multi edit dammi.io
Add titles, categories and tags with the Multi edit interface from Dammi.io.

Backend details

Behind the scenes the Dammi Daemons®, long running Symfony console commands, start working on the files.

Some of their jobs include:

  • Generating derivatives (thumbnails and the likes), normalising color profiles and stripping metadata using imagemagick.
  • Moving files from our on-site storage to S3.
  • Purging CDN caches.
  • Pushing file-metadata to redis (cache) and elasticsearch (search… obviously).
  • Permanently deleting files that have been in the trashcan for over seven days.

Files that have metadata attached to them will show up:

  • On the ‘recent files’ page.
  • When searched for.
  • In their respective categories.
  • Inside collections they were added to.

Dammi’s bottleneck

To keep every API consumer happy, we need to keep response time as low as possible.

Our main bottleneck turned out to be doctrine‘s ORM object hydration, which turns raw database results into php class instances and does data conversions (e.g.: unix timestamp => \DateTime). More importantly it keeps track of relationships between objects (e.g.: $file->addTag(new Tag("night out"));). This makes any CRUD operation a breeze… at the expense of performance though.

unix timestamp => \DateTime
 
$file->addTag(new Tag("night out"));

Especially in the case of ‘R’ where we are rarely operating on a single database row / object. The most obvious increase in performance can be gained from not lazy loading anything we know up front we are going to need. But this is only a micro-optimisation compared to the amount of CPU time and memory doctrine needs to hydrate these objects and all of their related entities. Which is why we chose to use redis to cache entire file objects.

We only do the absolute minimum, while our Daemons can process the more extensive operations.

A different approach could have been using the doctrine DBAL or a more lightweight hydrator, but we opted to bypass doctrine and mysql entirely wherever we could.

For all of the other letters of CRUD we do still use the doctrine ORM and optimized elsewhere.
As stated before, the Dammi Daemons do most of the heavy lifting. If a user wants to merge two tags that are attached to hundreds if not thousands of files, we don’t want him/her to wait for us to update the elastic index etc. So we only do the absolute minimum during the request while our Daemons can process the more extensive operations.

Roles in Dammi: embed and share

Dammi’s permission system is pretty simple, every role consists of the entity type and the HTTP verb along with a few more granular ones. The API does not keep track of users. Instead it requires consumers to send their API-keys and a token. This token is a timestamped hash, generated from of a set of roles, API-keys and secrets. These tokens should be generated server-side as they expire after fifteen minutes, while an API-key and secret pair would give a malicious geek access till the end of time.

The most restrictive role in Dammi (GET_PUBLIC_COLLECTION) will give you access to collections that have been shared, thus allowing you to embed a slideshow into your own website. Or sending a link to a media partner, allowing them to download a zip of your collection, without granting them any access to the rest of your media files.

Here’s a preview of how you can integrate a slideshow from Dammi, right into any site:

Graphical User Interface

The interface the users interact with, is built using Backbone and our javascript sdk (which abstracts the API consumption). We chose Backbone because we wanted to build a fast and flexible web application. Backbone is a rather small toolkit with some powerful classes and utilities that saves us a lot of boilerplate code and encourages devs to stick to a specific code style.

Our CSS preprocessor of choice SASS (+ Compass) and build system Gulp allow us to tailor the app to any of our clients’ needs. This ranges from a custom color and logo to extra metadata fields and different search behaviour.

Published by

Kobe Lipkens

Het technologisch brein achter de nationale stemtest, onze recentste webapp "Dammi" en nog vele andere zotte shit. Een greep uit zijn dagelijkse quotes: ait, stfu, dagani werke, nvm, ah cv, gvd, drupal suckt balle, nee peist ni, ...