Sharing secret between Gunicorn workers

 1. we found nice post from @jgleeee [link] [source code]; so we can use following to share object between workers.

  multiprocessing.Value

multiprocessing.Array

                multiprocessing.Manager

1.1. after digging, we found it's based on connection (pipe) across processes (it can operate across network). we think it's too complicated for a simple usage.

2. during digging into multiprocessing, we found shared memory which more match our scenario. [link]

2.1 based on SharedMemory we build following building block; 

# create
created = False
try:
    shared, created = SharedMemory(name="_shared_name", create=True, size=size), True
except FileExistsError:
    shared = SharedMemory(name="_shared_name")

# read

print(bytes(shared.buf))

# write
shared.buf[:size] = new_content[:size]

we now able to share secrets between process. but we found shared content can be simply fetched via

cat /dev/shm/_shared_name, CLEAR TEXT.

3. we need hide cleartext, choices are: 1. send across process, or 2. encrypt shared memory. obviously the choice is encryption.

3.1 need to share an encryption key between workers. this looks bring us back to memory sharing loop. but we found another way: gunicorn's pre_fork() is called before forking and in master process, post_fork() is called after forking and in worker process. so we build following building block.

def pre_fork(arbiter, worker):
    __secret = getattr(arbiter, '__secret', None)
    if not __secret:
        arbiter.__secret = os.urandom(16)

    worker.__secret = arbiter.__secret
    if len(arbiter.WORKERS) + 1 == workers:  # workers is final fork count 
        del arbiter.__secret


def post_fork(arbiter, worker):
    SECRET = worker.__secret
    del worker.__secret

after this, we shared static secret between workers.

3.2 connect static part and dynamic part

def has_data(obj, size):
    return bytes(obj.buf) != b'\x00' * size


def set_data(obj, size):
    obj.buf[:size] = aes.encrypt_cbc(data=data, key=SECRET)[:size]


def clear_data(obj, size):
    obj.buf[:size] = b'\x00' * size


def get_data(obj, size):
    return aes.decrypt_cbc(data=bytes(obj.buf), key=SECRET)

# ALL DONE