Convert ci-kubernetes-e2e-cri-gce-flaky to start with #1581

krzyzacy · 2017-01-18T00:22:20Z

split up #1569 #1525
ref #1225 #1226 #1230

This includes all the unit-test I'd like to add.
Pick an always-failing job to muck with :-)

krzyzacy · 2017-01-18T21:56:22Z

I0118 21:54:09.364] Call:  /workspace/./test-infra/jenkins/../jobs/pull-test-infra-bazel.sh
W0118 21:54:10.054] Could not open requirements file: [Errno 2] No such file or directory: 'gubernator/test_requirements.txt'
W0118 21:54:10.349] You are using pip version 8.1.1, however version 9.0.1 is available.
W0118 21:54:10.350] You should consider upgrading via the 'pip install --upgrade pip' command.
E0118 21:54:10.379] Build failed

spxtr · 2017-01-18T22:09:52Z

Yeah, I'm looking at that now.

krzyzacy · 2017-01-18T22:13:31Z

Wow and here it passes now...

fejta

Great start! Excited to see this get in.

fejta · 2017-01-18T22:25:39Z

scenarios/kubernetes_e2e.py

+
+    # Boilerplate envs
+    # Assume we're upping, testing, and downing a cluster
+    os.environ['E2E_UP'] = os.environ.get('E2E_UP') or 'true'


Consider dict.setdefault() for this

Also, I would be much happier if we simplify this to set the variables explicitly rather than relying on magic branching behavior.

I moved them to args.

fejta · 2017-01-18T22:31:59Z

scenarios/kubernetes_e2e.py

+        print >>sys.stderr, 'envfile %s does not exist' % envpath
+        return
+    with open(envpath, 'r') as envfile:
+        for line in continuation_lines(envfile):


This seems pretty fragile. I'd consider disallowing this pattern for the initial version. Also how are you handling quoting? Does it work with docker run --env-file?

fejta · 2017-01-18T22:34:04Z

scenarios/kubernetes_e2e.py

+    check(['docker', 'stop', CONTAINER])
+
+
+def sigint_handler(_signo, _frame):


Why do you have two functions that do exactly the same thing? Especially since whether it is handling a SIGTERM or SIGINT signal is captured in _signo.

Also prefixing method arguments with a underscore looks odd

fejta · 2017-01-18T22:36:12Z

scenarios/kubernetes_e2e.py

+      '-v', '/etc/localtime:/etc/localtime:ro'
+    ]
+
+    if os.environ.get('JENKINS_GCE_SSH_PRIVATE_KEY_FILE'):


To me this seems like something I would stick in a flag like we do elsewhere.

fejta · 2017-01-18T22:37:21Z

scenarios/kubernetes_e2e.py

+    PARSER = argparse.ArgumentParser(
+        'Runs e2e jobs on the kubernetes repo')
+    PARSER.add_argument(
+        '--branch', default='master', help='Upstream target repo')


Why does this exist?

fejta · 2017-01-18T22:44:43Z

scenarios/kubernetes_e2e.py

+    except subprocess.CalledProcessError as exc:
+        print 'Exiting with code: %d' % exc.returncode # fine?
+
+    signal.signal(signal.SIGTERM, sigterm_handler)


This waits until everything is complete before registering the signal handler, which means sigterm_handler will not catch the signal if one occurs curing check(*cmd) above.

fejta · 2017-01-18T22:45:10Z

scenarios/kubernetes_e2e.py

+        with open(e2e_image_tag_override, 'r') as tag:
+            e2e_image_tag = tag.read()
+
+    docker_env_ignore = [


I would much rather we switch to a whitelist rather than a blacklist.

Yeah agree. I'll make it a separate PR, I'd rather make a job works as previous first, and dump down what do we really need here.

fejta · 2017-01-18T22:46:06Z

scenarios/kubernetes_e2e.py

+        yield line
+
+
+def load_env(envpath):


Consider skipping this step entirely and pass --env-file directly to the docker run command.

I removed all not-yet-needed functions so it's cleaner now.

fejta · 2017-01-18T22:47:02Z

scenarios/kubernetes_e2e.py

+    artifacts = '%s/_artifacts' % workspace
+    os.makedirs(artifacts)
+
+    # TODO(ixdy): remove when all jobs are setting these vars using Jenkins credentials


Do we really need to keep this todo?

fejta · 2017-01-18T22:48:16Z

scenarios/kubernetes_e2e.py

+    PARSER.add_argument(
+        '--branch', default='master', help='Upstream target repo')
+    PARSER.add_argument(
+        '--platform', help='Platform test runs on')


Especially given that --platform is just a wrapper around loading a gce.env, to me the following seems a lot easier to understand:

kubernetes_e2e.py --env_file=gce.env --env_file=kubernetes-e2e-gce.env

In other words I would pretty much eliminate all of these flags. Especially things like --project and --zone which are specific to GCP

krzyzacy · 2017-01-18T23:57:20Z

@fejta updated, PTAL

krzyzacy · 2017-01-19T01:13:34Z

@k8s-bot bazel test this

fejta · 2017-01-19T01:02:27Z

scenarios/kubernetes_e2e.py

+        '--env-file',
+        action="append",
+        help='Job specific environment file')
+    PARSER.add_argument(


Will you match these flags to the kubernetes_kubelet.py scenario? (Update one or the other please)

https://github.com/kubernetes/test-infra/blob/master/scenarios/kubernetes_kubelet.py#L103

I edited the other one so I don't need include var function here.

fejta · 2017-01-19T01:04:03Z

scenarios/kubernetes_e2e.py

+
+    for key, value in os.environ.items():
+        if key not in docker_env_ignore:
+            cmd.extend(['-e', '%s=%s' % (key, value)])


I'm pretty sure this ordering regresses how things work: specifically if my environment has export INSTANCE_PREFIX=foo it ill now do the following:

docker run -e INSTANCE_PREFIX=bootstrap-e2e -e INSTANCE_PREFIX=foo

Previously the foo.sh would override INSTANCE_PREFIX=foo with INSTANCE_PREFXI=bootstrap-e2e and then we would only do the following:

env > env.list docker run --env-file=env.list # aka only one -e INSTANCE_PREFIX=bootstrape2e

Is this change in behavior intentional?

Oh, bah... I thought nobody is going to touch these boilerplates :-(

added a comment. this should ensure local var always overwrites anything else

fejta · 2017-01-19T01:09:25Z

scenarios/kubernetes_e2e.py

+    ]
+
+    docker_extra_args = []
+    if (os.environ.get('JENKINS_ENABLE_DOCKER_IN_DOCKER') and


It doesn't looks like the flaky job uses any of this functionality yet (overriding the runner, docker-in-docker, local binaries). Can we remove it from this PR?

deleted for now

fejta · 2017-01-19T01:12:15Z

scenarios/kubernetes_e2e.py

+
+    if args.env_file:
+        for env in args.env_file:
+            cmd.extend(['--env-file', env])


Suppose foo.env defines INSTANCE_PREFIX and/or KUBE_GKE_NETWORK, which is also set by --e2e-name. Which do you want to take precedence and why?

Then isn't it better to pre-process the envs like before then?

Since I can have FOO in .env file, and also set FOO locally.

fejta · 2017-01-19T01:13:26Z

scenarios/kubernetes_e2e.py

+    ])
+
+    try:
+        signal.signal(signal.SIGTERM, sig_handler)


Minimize the amount of code inside a try block. Move the signal registration (which will never raise a CalledProcessError) outside of it:

https://google.github.io/styleguide/pyguide.html?showone=Exceptions#Exceptions

fejta · 2017-01-19T01:13:51Z

scenarios/kubernetes_e2e.py

+    PARSER.add_argument(
+        '--e2e-name',
+        default='bootstrap-e2e',
+        help='If we need to set --down in e2e.go')


Copy-paste error

fejta · 2017-01-19T01:14:41Z

scenarios/kubernetes_e2e.py

+    os.environ['PATH'] += ':/usr/local/go/bin'
+
+    # dockerized-e2e-runner goodies setup
+    workspace = os.environ.get('WORKSPACE') or os.getcwd()


setdefault?

something like this?

fejta · 2017-01-19T01:15:10Z

scenarios/kubernetes_e2e.py

+def main(args):
+    """Set up env, trigger runner, handle termination. """
+    # pylint: disable=too-many-locals
+    # pylint: disable=too-many-branches


Please enable too-many-branches (the other two are fine). I would not expect this method to have an excessive number of branches.

fejta · 2017-01-19T01:16:27Z

scenarios/kubernetes_e2e.py

+    gce_pub = '%s.pub' % gce_ssh
+    service = '/service-account.json'
+
+    cmd.extend(['-v', '%s:%s:ro' % (args.gce_ssh, gce_ssh)])


Any reason not to combine all these extends into into the cmd = [] above?

I suspect that will enable you to enable the too-many-statements check

krzyzacy · 2017-01-19T18:46:19Z

# pylint: disable=duplicate-code seems has no effect :-(

pylint-dev/pylint#214

krzyzacy · 2017-01-19T19:13:04Z

@fejta PTAL again

fejta · 2017-01-19T19:46:53Z

scenarios/kubernetes_e2e.py

+        help='Job specific environment file')
+
+    PARSER.add_argument(
+        '--gce-ssh',


What I mean is that this flag is called --ssh in the kubelet and --gce-ssh in the e2e scenarios. I would like the flag names to be the same. Please call both of them the same thing.

updated kubelet, since here later we will need aws-ssh and aws-pub

fejta · 2017-01-19T19:49:10Z

scenarios/kubernetes_e2e.py

+    gce_pub = '%s.pub' % gce_ssh
+    service = '/service-account.json'
+
+    cmd.extend(['-v', '%s:%s:ro' % (args.gce_ssh, gce_ssh),


It still seems like these 7 cmd.extend() commands can be combined.

fejta · 2017-01-19T19:50:28Z

scenarios/kubernetes_e2e.py

+        if key not in docker_env_ignore:
+            cmd.extend(['-e', '%s=%s' % (key, value)])
+
+    cmd.extend([


Is there a reason these two need to go at the end instead of with the other -e stanzas?

Looks like you can now switch this to cmd.append() 😁

fejta · 2017-01-19T19:50:45Z

scenarios/kubernetes_e2e.py

+        check(*cmd)
+        print 'Finished Successfully'
+    except subprocess.CalledProcessError as exc:
+        print 'Exiting with code: %d' % exc.returncode # fine?


What does the fine? comment mean?

Also, consider writing errors to stderr instead of stdout

done. fine is my best wish to the failing test...

fejta · 2017-01-19T19:53:22Z

scenarios/kubernetes_e2e.py

+
+    lookup = {'workspace':os.environ.get('WORKSPACE')}
+
+    # Get golang into our PATH so we can run e2e.go


I do not understand this

ref from https://github.com/krzyzacy/test-infra/blob/master/jobs/ci-kubernetes-e2e-cri-gce-flaky.sh#L68-L69

Yes but why is it there? What is it trying to do? If you are changing os.environ but not using it directly it is a pretty strong signal to think about what it is trying to do and if there is a better way to do it.

It does not seem to do anything since PATH is specifically not mapped into docker? From the comment I would assume it was some legacy at some point?

fejta · 2017-01-19T19:57:33Z

scenarios/kubernetes_e2e.py

+    lookup = {'workspace':os.environ.get('WORKSPACE')}
+
+    # Get golang into our PATH so we can run e2e.go
+    os.environ['PATH'] += ':/usr/local/go/bin'


nit: consider using % instead of +=

fejta · 2017-01-19T19:58:29Z

scenarios/kubernetes_e2e.py

+    # dockerized-e2e-runner goodies setup
+    workspace = lookup.setdefault('workspace', os.getcwd())
+    artifacts = '%s/_artifacts' % workspace
+    os.makedirs(artifacts)


This will fail if $WORKSPACE/_artifacts already exists, unlike mkdir -p

https://docs.python.org/2/library/os.html#os.makedirs

added check.

fejta · 2017-01-19T19:59:03Z

scenarios/kubernetes_e2e.py

+    e2e_image_tag = 'v20170104-9031f1d'
+    e2e_image_tag_override = '%s/hack/jenkins/.kubekins_e2e_image_tag' % workspace
+    if os.path.isfile(e2e_image_tag_override):
+        with open(e2e_image_tag_override, 'r') as tag:


nit: 'r' is unnecessary as this is the default

fejta · 2017-01-19T20:01:53Z

scenarios/kubernetes_e2e.py

+    # Use default component update behavior
+    cmd.extend(['-e', 'CLOUDSDK_EXPERIMENTAL_FAST_COMPONENT_UPDATE=\'false\''])
+
+    cmd.extend(['-e', 'E2E_UP=%s' % args.e2e_up,


nit: if you're using multiple lines please do a line break beforehand so that we do not need to adjust things if you change the name of cmd to commmand.

Aka if you do this:

change_cmd.extend(['if I change this line', 'Note how this line is now misaligned']) change_cmd.extend([ 'However with this strategy', 'the padding does not change when I change the first line'])

fejta · 2017-01-19T20:03:35Z

scenarios/kubernetes_e2e.py

+    try:
+        check(*cmd)
+        print 'Finished Successfully'
+    except subprocess.CalledProcessError as exc:


This file needs to raise and exception and/or exit 1 whenever tests fail. Right now main() will always exit 0 regardless of whether tests fail.

I was once intended to trap return signal from exc.returncode.

Maybe I can get rid of this try block all together just call check(*cmd)

raises the exception and let upper layer deals with it.

k8s-ci-robot · 2017-01-19T20:28:55Z

Bazel test failed for commit 7a7bbf2. Full PR test history. cc @krzyzacy

The magic incantation to run this job again is @k8s-bot bazel test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

krzyzacy · 2017-01-19T20:55:59Z

another iteration, comments addressed.

fejta · 2017-01-19T20:58:11Z

scenarios/kubernetes_e2e.py

+        print 'Finished Successfully'
+    except subprocess.CalledProcessError as exc:
+        print >>sys.stderr, 'Exiting with code: %d' % exc.returncode
+        raise exc


Please just use raise to preserve the stack trace

Compare:

def Raises(): raise ValueError('hello there') def CatchAndRaise(): try: Raises() except ValueError as ve: raise ve def OnlyRaise(): try: Raises() except ValueError as ve: raise

I got rid of this try-catch

fejta · 2017-01-19T21:00:58Z

scenarios/kubernetes_e2e.py

+    """Set up env, trigger runner, handle termination. """
+    # pylint: disable=too-many-locals
+
+    lookup = {}


Again, why do you need this variable? It doesn't seem to serve a useful purpose

ehhhh isn't it I need to declare a dict first?

I am pretty sure these two things do the same thing:

lookup = 5 lookup += alpha workspace = lookup + beta # lookup is pointless, get rid of it workspace = 5 + alpha + beta # same result

ehhh then how do I call setdefault?

Why do you need to call that?

fejta · 2017-01-19T21:05:38Z

scenarios/kubernetes_e2e.py

+
+    # Assume we're upping, testing, and downing a cluster by default
+    PARSER.add_argument(
+        '--e2e-up', default='true', help='If we need to set --up in e2e.go')


Given that the scenario is kubernetes_e2e.py it seems redundant to prefix flags with --e2e, especially since not all the flags have this prefix.

fejta · 2017-01-19T21:06:18Z

scenarios/kubernetes_e2e.py

+    PARSER.add_argument(
+        '--e2e-down', default='true', help='If we need to set --down in e2e.go')
+    PARSER.add_argument(
+        '--e2e-name', default='bootstrap-e2e', help='Name of the e2e test')


How about --cluster since the primary purpose of this is to name the cluster.

fejta · 2017-01-19T21:08:15Z

jenkins/bootstrap_test.py

+                    realjob = self.GetRealBootstrapJob('job-configs/kubernetes-jenkins/bootstrap-ci.yaml', job)
+                    self.assertTrue(realjob)
+                    docker_timeout = realjob['timeout']
+                    print docker_timeout


Is this left over debug code? Please assert something here, rather than print.

fejta · 2017-01-19T21:10:10Z

jenkins/job-configs/kubernetes-jenkins/bootstrap-ci.yaml

@@ -51,413 +53,605 @@
    - kubernetes-e2e-aws:
        job-name: ci-kubernetes-e2e-aws
        frequency: 'H/5 * * * *' # At least every 30m
+        timeout: 140


For jobs you aren't changing please leave the timeout at 0.

Also given the enormous number of jobs you're changing, will you send me a separate PR that adds --json='{json}' --timeout='{timeout}' to the template, and then sets these values to zero to all jobs?

fejta · 2017-01-19T21:13:43Z

jenkins/bootstrap_test.py

+                self.assertTrue('scenario' in config[job], job)
+                self.assertTrue(os.path.isfile(bootstrap.test_infra('scenarios/%s.py' % config[job]['scenario'])), job)
+                if 'args' in config[job]:
+                    for arg in config[job].get('args', []):


The if statement makes the get('args', []) redundant. Please only keep one of these

fejta · 2017-01-19T21:14:28Z

jenkins/bootstrap_test.py

+                self.assertTrue(os.path.isfile(bootstrap.test_infra('scenarios/%s.py' % config[job]['scenario'])), job)
+                if 'args' in config[job]:
+                    for arg in config[job].get('args', []):
+                        m = re.match(r'--env-file=([^\"]+)', arg)


If the scenario is kubernetes-e2e I would validate that there is at least one --env-file flag

fejta · 2017-01-19T21:14:55Z

jenkins/bootstrap_test.py

@@ -1527,6 +1581,19 @@ def testNoBadVarsInJobs(self):
            if bad_vars:
                self.fail('Job %s contains bad bash variables: %s' % (job, ' '.join(bad_vars)))

+    def testValidJobEnvs(self):


Fantastic unit test!

fejta · 2017-01-19T21:37:54Z

jenkins/bootstrap_test.py

@@ -1403,10 +1409,33 @@ def CheckBootstrapYaml(self, path, check, suffix='suffix', use_json=False):
            self.assertTrue(job_name)
            self.assertEquals(job_name, real_job.get('job-name'))

+    def GetRealBootstrapJob(self, path, job):


Should CheckBootstrapYaml use this function? It seems to duplicate a lot of functionality.

IMO This is a dump down version and 'assumes' everything would work without error after passes CheckBootstrapYaml (which has all the assertions)..

It seems slow and silly to load the yaml file twice

Oh indeed it will load the yaml n times :-(... I'll do something here

how about something like this

krzyzacy · 2017-01-19T22:58:48Z

@fejta updated

fejta

Looking really good. This is really close. Just a couple nits and then would like to do a little bit more on the realjobs situation.

fejta · 2017-01-20T02:01:15Z

scenarios/kubernetes_e2e.py

+        if key not in docker_env_ignore:
+            cmd.extend(['-e', '%s=%s' % (key, value)])
+
+    cmd.extend([


Looks like you can now switch this to cmd.append() 😁

fejta · 2017-01-20T02:02:14Z

scenarios/kubernetes_e2e.py

+
+
+def sig_handler(_signo, _frame):
+    """Handle signal.SIGTERM and signal.SIGINT."""


Noting that this stops the container would be more useful documentation.

fejta · 2017-01-20T02:02:52Z

scenarios/kubernetes_e2e.py

+
+
+def main(args):
+    """Set up env, trigger runner, handle termination. """


nit: starts kubekins-e2e may be a better description than trigger runner.

fejta · 2017-01-20T02:03:23Z

platforms/gke.env

@@ -0,0 +1,8 @@
+# GKE E2E ENVS


Not needed by this pr, please remove

fejta · 2017-01-20T02:06:54Z

jenkins/bootstrap_test.py

+                    mat = re.match(r'export KUBEKINS_TIMEOUT="(\d+)m".*', line)
+                else:
+                    mat = re.match(r'KUBEKINS_TIMEOUT="(\d+)m".*', line)
+                    realjob = self.GetRealBootstrapJob(job)


This doesn't seem like a super great strategy either. Specifically you're now depending on the fact that testValidTimeout runs after testBootstrapCIYaml based on where they are defined in this file, which does not seem like a great practice.

fejta · 2017-01-20T02:08:17Z

jenkins/bootstrap_test.py

@@ -1416,10 +1426,16 @@ def CheckBootstrapYaml(self, path, check, suffix='suffix', use_json=False):
            self.assertTrue(job_name)
            self.assertEquals(job_name, real_job.get('job-name'))

+    def GetRealBootstrapJob(self, job):
+        key = os.path.splitext(job.strip())[0][3:]


Consider refactoring this to check and see whether key is in self.realjobs, returning the cached version if it is, otherwise loading the yaml. Then update CheckBootstrapYaml() to call this.

Ohhhh... the issue here is that from the job I would not know which yaml file to load

Pass in job_path as well

job_path is going to be path/to/jobs/FOO.sh rather than bootstrap-foo.yaml, and jobs are gathered from config.json and iter all files from jobs/ dir. I can list out all the existing yaml files and load them, if a job is not cached yet?

Sounds like it will be easier to put it in the yaml test then?

Something like this? It works if I move testValidTimeout to the front.

fejta · 2017-01-20T22:41:06Z

LGTM aside from the GetRealBootstrapJob fix

fejta · 2017-01-21T02:17:41Z

/lgtm

krzyzacy · 2017-01-21T03:32:21Z

let the fire burn!

…r images as necessary. (kubernetes#1581)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 18, 2017

krzyzacy assigned fejta Jan 18, 2017

krzyzacy force-pushed the e2e-split branch 4 times, most recently from 1a7f970 to e7b8d33 Compare January 18, 2017 21:53

krzyzacy closed this Jan 18, 2017

krzyzacy reopened this Jan 18, 2017

fejta reviewed Jan 18, 2017

View reviewed changes

krzyzacy force-pushed the e2e-split branch from e7b8d33 to 26dfbf5 Compare January 18, 2017 23:54

krzyzacy force-pushed the e2e-split branch 2 times, most recently from 2358abf to 07bb4b6 Compare January 19, 2017 00:10

krzyzacy closed this Jan 19, 2017

krzyzacy reopened this Jan 19, 2017

fejta reviewed Jan 19, 2017

View reviewed changes

krzyzacy force-pushed the e2e-split branch 5 times, most recently from f6d7197 to 322e6e3 Compare January 19, 2017 02:28

krzyzacy force-pushed the e2e-split branch 2 times, most recently from 0afe01c to 2194bb7 Compare January 19, 2017 18:58

fejta reviewed Jan 19, 2017

View reviewed changes

krzyzacy force-pushed the e2e-split branch from 2194bb7 to 7a7bbf2 Compare January 19, 2017 20:27

krzyzacy force-pushed the e2e-split branch 4 times, most recently from 5d838f3 to 6eb2916 Compare January 19, 2017 20:51

fejta reviewed Jan 19, 2017

View reviewed changes

krzyzacy mentioned this pull request Jan 19, 2017

Init timeout and json to 0, and use this flag, for e2e jobs #1602

Merged

krzyzacy force-pushed the e2e-split branch 2 times, most recently from 8593c52 to 35e3f9a Compare January 19, 2017 22:54

fejta reviewed Jan 20, 2017

View reviewed changes

krzyzacy force-pushed the e2e-split branch from 35e3f9a to 18065c9 Compare January 20, 2017 19:56

convert ci-kubernetes-e2e-cri-gce-flaky to start with

93b31cd

krzyzacy force-pushed the e2e-split branch from 18065c9 to 93b31cd Compare January 20, 2017 23:53

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 21, 2017

krzyzacy merged commit 901411f into kubernetes:master Jan 21, 2017

krzyzacy deleted the e2e-split branch January 21, 2017 03:32

grantr pushed a commit to grantr/test-infra that referenced this pull request Feb 21, 2020

Update prow from v20191128-48349ea19 to v20191204-d7993c80e, and othe…

71f5202

…r images as necessary. (kubernetes#1581)

		check(['docker', 'stop', CONTAINER])


		def sigint_handler(_signo, _frame):


		lookup = {'workspace':os.environ.get('WORKSPACE')}

		# Get golang into our PATH so we can run e2e.go



		def sig_handler(_signo, _frame):
		"""Handle signal.SIGTERM and signal.SIGINT."""



		def main(args):
		"""Set up env, trigger runner, handle termination. """

Convert ci-kubernetes-e2e-cri-gce-flaky to start with #1581

Convert ci-kubernetes-e2e-cri-gce-flaky to start with #1581

Uh oh!

Conversation

krzyzacy commented Jan 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krzyzacy commented Jan 18, 2017

Uh oh!

spxtr commented Jan 18, 2017

Uh oh!

krzyzacy commented Jan 18, 2017

Uh oh!

fejta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krzyzacy commented Jan 18, 2017

Uh oh!

krzyzacy commented Jan 19, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krzyzacy commented Jan 18, 2017 •

edited

Loading