MongoDB select fields after $lookup

When there is a $lookup stage to join a list of large documents, an error Total size of documents in ... matching ... exceeds maximum document size can arrive.

It’s possible to avoid this with $unwind stage right after $lookup. More explanations in the documentation. And then the documents can be regrouped with the required fields.

Order.objects.aggregate(
    {
        '$lookup': {
            'from': 'item',
            'localField': '_id',
            'foreignField': 'order_id',
            'as': 'items'
        }
    },
    {
        "$unwind": "$items"
    },
    {
        "$group": {
            "_id": "$_id",
            "date": {"$last": "$date"},
            "items": {
                "$push": {
                    "name": "$items.name",
                    "price": "$items.price"
                }
            }
        }
    }
)

How to find a change for a field with MongoDB aggregation

For example there is a collection device_status which stores the different states for the devices. The task is to find the devices which passed from off to on at least one time.

{ "device" : "device1", "state" : "on", "ts": ISODate("2018-06-07T17:05:29.340+0000") }
{ "device" : "device2", "state" : "off", "ts": ISODate("2018-06-08T17:05:29.340+0000") }
{ "device" : "device3", "state" : "on", "ts": ISODate("2018-06-09T17:05:29.340+0000")}
{ "device" : "device3", "state" : "shutdown", "ts": ISODate("2018-06-09T18:05:29.340+0000")}
{ "device" : "device2", "state" : "load", "ts": ISODate("2018-06-09T19:05:29.340+0000") }
{ "device" : "device2", "state" : "on", "ts": ISODate("2018-06-10T17:05:29.340+0000") }
{ "device" : "device3", "state" : "off", "ts": ISODate("2018-06-11T17:05:29.340+0000") }
{ "device" : "device1", "state" : "idle", "ts": ISODate("2018-06-11T18:05:29.340+0000") }
{ "device" : "device3", "state" : "on", "ts": ISODate("2018-06-12T17:05:29.340+0000") }
...

The first stage is to sort the data by device and date.

A bootstrap for a microservice based on Flask with MongoDB

Starting a new project is a common task in microservices architecture. To do this it’s better to have a some template. I put my version to the flask-mongoengine-bootstrap repository. The key point are:

  • very basic, only flask, flask-mongoengine and structlog in requirements
  • configuration through environment variables
  • configured logging in JSON format
  • marking log records with request_id
  • possibility to run development version make dev and tests make test through docker
  • a template for Makefile
  • examples of model, api route and tests

If you’ll use it, do not forget to change SECRET_KEY.

A simple CRUD app with Django and Mongoengine

| Comments

There are several possibilities to use MongoDB in Django:

Django MongoDB uses django-nonrel, which is a fork based on Django 1.3.
I don’t like this idea, because now Django 1.5 is ready to out. Beetween Mongoengine and MongoKit, I like more Mongoengine. There are several comparative articles:

So I created a simple CRUD app using Mongoengine. The model definition in Mongoengine looks like in Django.