new redis toolset to use with web2py

This is a refactor of everything web2py uses with redis.
Specifically:
- a refactored redis_cache.py that fixes #958, allowing a "fail
gracefully" behaviour in case redis is not available
- a refactored redis_session.py that blocks less with with_lock=True
(although still not optimal)
- a new (and NEEDED) redis_utils.py that serves as the base for
everything else, allowing an RConn object that you can freely use as a
redis.StrictRedis connection, and that you can override in case you're
using a different library (or that web2py will use in case redis-py
won't be the de-facto standard around)
- a newly - and much anticipated - redis_scheduler.py. It's a slip-in
replacement for the standard scheduler that uses redis for workers
coordination. Feel free to dig in the code and improve it.

For redis_cache and redis_session changes are BREAKING. It means that
users will need to change the import locations and tune a bit the code.
Now every module depends on an gluon.contrib.redis_utils.RConn object
(or similar) that in turns is very similar to a redis.StrictRedis one.
The redis instance is EXTERNAL to the modules themselves (no more "host,
port, db, password" parameters in RedisCache or RedisSession)
See the relevant docstrings for usage examples
This commit is contained in:
niphlod committed 2016-01-02 23:15:00 +01:00
1 parent 1ca0c9b0c0
commit 12acdb51d7
5 files changed
+980 -128

No files matched your search

+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Developed by niphlod@gmail.com
License MIT/BSD/GPL
Serves as base to implement Redis connection object and various utils
for redis_cache, redis_session and redis_scheduler in the future
Should-could be overriden in case redis doesn't keep up (e.g. cluster support)
to ensure compatibility with another - similar - library
"""
import logging
import thread
import time
from gluon import current
logger = logging.getLogger("web2py.redis_utils")
try:
import redis
from redis.exceptions import WatchError as RWatchError
from redis.exceptions import ConnectionError as RConnectionError
except ImportError:
logger.error("Needs redis library to work")
raise RuntimeError('Needs redis library to work')
locker = thread.allocate_lock()
def RConn(*args, **vars):
"""
Istantiates a StrictRedis connection with parameters, at the first time
only
"""
locker.acquire()
try:
instance_name = 'redis_conn_' + current.request.application
if not hasattr(RConn, instance_name):
setattr(RConn, instance_name, redis.StrictRedis(*args, **vars))
return getattr(RConn, instance_name)
finally:
locker.release()
def acquire_lock(conn, lockname, identifier, ltime=10):
while True:
if conn.set(lockname, identifier, ex=ltime, nx=True):
return identifier
time.sleep(.01)
_LUA_RELEASE_LOCK = """
if redis.call("get", KEYS[1]) == ARGV[1]
then
return redis.call("del", KEYS[1])
else
return 0
end
"""
def release_lock(instance, lockname, identifier):
return instance._release_script(
keys=[lockname], args=[identifier])
def register_release_lock(conn):
rtn = conn.register_script(_LUA_RELEASE_LOCK)
return rtn