How to connect to a service in docker swarm with docker

docker, swarm | Comments

It’s a rare case, suppose there is a myapp swarm cluster with myapp_mongo database without published port and there is a need to run a command from some docker image with connection to this database.

By default docker stack deploy creates a non-attachable network, so docker run --network myapp_default will output Error response from daemon: Could not attach to network myapp_default: rpc error: code = PermissionDenied desc = network myapp_default not manually attachable.

A way to bypass it is to create a new attachable network and attach it to the service.

docker network create --driver overlay --attachable mongo_network
docker service update --network-add mongo_network myapp_mongo
docker run --rm --network mongo_network mongo:4.0.6 mongodump -h lsicloud_mongo ...
docker service update --network-rm mongo_network lsicloud_mongo
docker network rm mongo_network

Get blinker signal's receivers names

python | Comments 
from unittest import TestCase
from app.main import init_app
from app.signals import my_signal


class SignalTests(TestCase):

    def setUp(self):
        init_app()

    def test_signal_is_connected_to_my_receiver(self)
        receivers = [r().__name__ for r in signal.receivers.values()]
        self.assertIn('my_receiver', receivers)

MongoDB dump and restore with docker

mongodb, docker | Comments

Here are two commands to take a partial dump of the collection from production database and put it in dev mongo instance running through docker-compose.

docker run -v `pwd`/:/dump mongo mongodump --gzip --archive=/dump/my_collection.agz --host <connection url> --ssl --username <username> --password <password> --authenticationDatabase admin --db <prod_db> --collection my_collection --query '{date: {$gte:  ISODate("2019-02-01T00:00:00.000+0000")}}'

docker-compose run -v `pwd`/my_collection.agz:/my_collection.agz mongo mongorestore --gzip --archive=/my_collection.agz --host mongo --nsFrom <prod_db>.my_collection --nsTo <dev_db>.my_collection

Easy charts in python app with plotly

python, plotly | Comments

An example how to add a chart to Flask app with plotly library:

def get_chart_data():
    return mongo_db['item'].aggregate([
        {'$sort': {'date': 1}},
        {'$group': {
            '_id': {'year': {'$year': '$date'}, 'month': {'$month': '$date'}},
            'num': {'$sum': 1}
        }},
        {'$project': {
            'date': {'$dateFromParts' : {'year': '$_id.year', 'month': '$_id.month'}},
            'num': '$num'
        }},
        {'$sort': {'date': 1}}
    ])


def get_chart():
    data = list(get_chart_data())
    layout = plotly.graph_objs.Layout(title='Items by month')
    scatter_data = [
        plotly.graph_objs.Scatter(
            x=[d['date'] for d in data], 
            y=[d['num'] for d in data]
        )
    ]
    fig = plotly.graph_objs.Figure(data=scatter_data, layout=layout)
    return plotly.offline.plot(fig, include_plotlyjs=True, output_type='div')


@route('/chart')
def chart():
    return render_template('chart.html', chart=get_chart())

Jinja template:

{{ chart|safe }}

For jupyter notebook it will be:

from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
iplot(get_chart())

Mongoengine as_pymongo performance

python, mongo, mongoengine | Comments

When you need to get only several fields from the list of complex objects, it works much faster with as_pymongo function.

In my situation I have a 16x increase:

%timeit list(SomeObject.objects.scalar('id')[0:500])
# 129 ms ± 11 ms per loop

%timeit list(o['_id'] for o in SomeObject.objects.scalar('id')[0:500].as_pymongo())
# 7.98 ms ± 849 µs per loop

How to find the document with maximum size in MongoDB collection

mongodb | Comments

The command to get document size is Object.bsonsize. The next query is to find the document in a small collection, cause it can be slow:

db.getCollection('my_collection').find({}).map(doc => {
    return {_id: doc._id, size: Object.bsonsize(doc)};
}).reduce((a, b) => a.size > b.size ? a : b)

To do this faster with mongo mapReduce:

db.getCollection('my_collection').mapReduce(
    function() {
        emit('size', {_id: this._id, size: Object.bsonsize(this)});
    },
    function(key, values) {
        return values.reduce((a, b) => a.size > b.size ? a : b);
    },
    {out: {inline: 1}}
)

How to find number of MongoDB connections

mongodb | Comments

From the MongoDB side the current connections can be found with db.currentOp() command. Then they can be grouped by client ip, counted and sorted.

var ips = db.currentOp(true).inprog.filter(
        d => d.client
    ).map(
        d => d.client.split(':')[0]
    ).reduce(
        (ips, ip) => {
            if(!ips[ip]) {
                ips[ip] = 0;
            }
            ips[ip]++;
            return ips;
        }, {}
    );
Object.keys(ips).map(
        key => {
            return {"ip": key, "num": ips[key]};
        }
    ).sort(
        (a, b) => b.num - a.num
    );

The result will be like this:

[
    {
        "ip" : "11.22.33.444",
        "num" : 77.0
    },
    {
        "ip" : "11.22.33.445",
        "num" : 63.0
    },
    {
        "ip" : "11.22.33.344",
        "num" : 57.0
    }
]

Then if there are several Docker containers on client host, the connections can be found by netstat command in each of them. Suppose there are several MongoDB replicas with ips starting on 44.55... and 77.88..., the command to count all connections to the replicas is:

netstat -tn | grep -e 44.55 -e 77.88 | wc -l

Maximum number of client connections in Flask-SocketIO with Eventlet

flask, python, socketio | Comments

It’s not mentioned in the docs for Flask-SocketIO that Eventlet has an option max_size which by default limits the maximum number of client connections opened at any time to 1024. There is no way to pass it through flask run command, so the application should be run with socketio.run, for example:

...
if __name__ == '__main__':
    socketio.run(app, host='0.0.0.0', port='8080', max_size=int(os.environ.get('EVENTLET_MAX_SIZE', 1024)))
« 4/13 »

© 2024 Andrey Zhukov's Tech Blog.