Add cachecontrol to requests to github

Github is very strict about quotas per authentication. To prevent going
over these quotas and to ensure we don't get marked as spam we should
make sure to respect the Etags present in responses.

Use cachecontrol to give us a caching layer that will fullfil requests
that are cachable without going over the network.

A point to consider here is that there does not appear to be a way to
vary caching based on the current authentication - so any per auth
requests may be mishandled or auth may expire. I don't think there is
any concern here as it's simply zuul making the requests.

Change-Id: I04bfc0cfec1ffc8ebdfd2d9181ac3119cc6e14ac
Signed-off-by: Jamie Lennox <jamielennox@gmail.com>

requirements.txt

@@ -23,3 +23,4 @@ kazoo
 sqlalchemy
 alembic
 cryptography>=1.6
+cachecontrol

zuul/driver/github/githubconnection.py

@@ -18,6 +18,8 @@ import hmac
 import hashlib
 import time
 
+import cachecontrol
+from cachecontrol.cache import DictCache
 import webob
 import webob.dec
 import voluptuous as v
@@ -291,6 +293,13 @@ class GithubConnection(BaseConnection):
             'canonical_hostname', self.git_host)
         self.source = driver.getSource(self)
 
+        # NOTE(jamielennox): Better here would be to cache to memcache or file
+        # or something external - but zuul already sucks at restarting so in
+        # memory probably doesn't make this much worse.
+        self.cache_adapter = cachecontrol.CacheControlAdapter(
+            DictCache(),
+            cache_etags=True)
+
     def onLoad(self):
         webhook_listener = GithubWebhookListener(self)
         self.registerHttpHandler(self.payload_path,
@@ -309,6 +318,10 @@ class GithubConnection(BaseConnection):
             else:
                 self.github = github3.login(token=token)
             self.log.info("Github API Authentication successful.")
+
+            # anything going through requests to http/s goes through cache
+            self.github.session.mount('http://', self.cache_adapter)
+            self.github.session.mount('https://', self.cache_adapter)
         else:
             self.github = None
             self.log.info(