WEBVTT

00:00:00.000 --> 00:00:03.465 align:middle line:90%
[MUSIC PLAYING]

00:00:03.465 --> 00:00:16.830 align:middle line:90%


00:00:16.830 --> 00:00:21.000 align:middle line:84%
Continuous testing has a place
in any DevOps environment.

00:00:21.000 --> 00:00:24.960 align:middle line:84%
We want to talk now about how
to get your tests integrated

00:00:24.960 --> 00:00:26.590 align:middle line:90%
with your deployment pipeline.

00:00:26.590 --> 00:00:29.550 align:middle line:84%
So we can test at
every level that we

00:00:29.550 --> 00:00:33.390 align:middle line:84%
need to get high quality
code and confidence into what

00:00:33.390 --> 00:00:34.230 align:middle line:90%
we're shipping.

00:00:34.230 --> 00:00:36.360 align:middle line:90%
So what is continuous testing?

00:00:36.360 --> 00:00:39.210 align:middle line:84%
It's not a technical term that
you might find in industry

00:00:39.210 --> 00:00:41.790 align:middle line:84%
everywhere, but it's
a term that connotes

00:00:41.790 --> 00:00:43.150 align:middle line:90%
what we're looking for.

00:00:43.150 --> 00:00:47.640 align:middle line:84%
It's this execution of tests
repeatedly against a code base

00:00:47.640 --> 00:00:50.580 align:middle line:90%
and a deployment environment.

00:00:50.580 --> 00:00:53.700 align:middle line:84%
And it can be automated unit
tests, integration tests,

00:00:53.700 --> 00:00:55.830 align:middle line:90%
coded UI tests--

00:00:55.830 --> 00:00:57.630 align:middle line:90%
any type of testing.

00:00:57.630 --> 00:01:01.590 align:middle line:84%
And the depth of testing often
progresses as the environment

00:01:01.590 --> 00:01:02.640 align:middle line:90%
gets closer to prod.

00:01:02.640 --> 00:01:05.550 align:middle line:84%
So for instance, we might
be running unit tests

00:01:05.550 --> 00:01:09.270 align:middle line:84%
very quickly as we're doing our
continuous integration builds.

00:01:09.270 --> 00:01:12.090 align:middle line:84%
And as we get closer
to production,

00:01:12.090 --> 00:01:14.730 align:middle line:84%
some of the longer
running load tests--

00:01:14.730 --> 00:01:18.480 align:middle line:84%
which might run for several
days or at least several hours--

00:01:18.480 --> 00:01:22.230 align:middle line:84%
are things that we probably
won't run on every deployment

00:01:22.230 --> 00:01:24.130 align:middle line:84%
to lower levels of
the environment.

00:01:24.130 --> 00:01:26.880 align:middle line:84%
So we might progressively
add more tests

00:01:26.880 --> 00:01:30.420 align:middle line:84%
and more capabilities as
we move closer and closer

00:01:30.420 --> 00:01:31.350 align:middle line:90%
to production.

00:01:31.350 --> 00:01:33.240 align:middle line:84%
Now, do you run
any tests in prod?

00:01:33.240 --> 00:01:35.820 align:middle line:84%
Absolutely, your build
verification tests,

00:01:35.820 --> 00:01:36.873 align:middle line:90%
your smoke tests.

00:01:36.873 --> 00:01:38.790 align:middle line:84%
There's a lot of things
you might run in prod,

00:01:38.790 --> 00:01:41.520 align:middle line:84%
but that also is where a
lot of your instrumentation

00:01:41.520 --> 00:01:42.520 align:middle line:90%
will kick in.

00:01:42.520 --> 00:01:48.120 align:middle line:84%
So in practise, continuous
testing is often one

00:01:48.120 --> 00:01:52.860 align:middle line:84%
of the most difficult and time
consuming pieces to get right.

00:01:52.860 --> 00:01:54.750 align:middle line:84%
It's often one of
the later things

00:01:54.750 --> 00:01:58.920 align:middle line:84%
that people plug in to their
DevOps environment, and why?

00:01:58.920 --> 00:02:02.010 align:middle line:84%
Because as your
application evolves,

00:02:02.010 --> 00:02:04.290 align:middle line:84%
and as your
infrastructure evolves,

00:02:04.290 --> 00:02:08.729 align:middle line:84%
and as your configuration
evolves, all of these things

00:02:08.729 --> 00:02:11.190 align:middle line:84%
can break existing
tests and cause

00:02:11.190 --> 00:02:12.900 align:middle line:90%
them to need to be rewritten.

00:02:12.900 --> 00:02:16.650 align:middle line:84%
So you might need to spend
a lot of your time getting

00:02:16.650 --> 00:02:18.060 align:middle line:90%
these tests just right.

00:02:18.060 --> 00:02:19.170 align:middle line:90%
Is it worth it?

00:02:19.170 --> 00:02:20.430 align:middle line:90%
Absolutely.

00:02:20.430 --> 00:02:22.440 align:middle line:84%
So what are some of
the types of testing?

00:02:22.440 --> 00:02:25.060 align:middle line:84%
And this is, by all means,
not an exhaustive list,

00:02:25.060 --> 00:02:27.630 align:middle line:84%
but it does bring us
from the simplest test

00:02:27.630 --> 00:02:30.360 align:middle line:84%
that we might run early
to some of the later tests

00:02:30.360 --> 00:02:31.410 align:middle line:90%
that we'll run late.

00:02:31.410 --> 00:02:35.880 align:middle line:84%
Unit testing, testing of
a system in isolation,

00:02:35.880 --> 00:02:38.880 align:middle line:84%
those things run very quickly
on the order of milliseconds.

00:02:38.880 --> 00:02:41.970 align:middle line:84%
And you'll run these quite
often, usually at every build.

00:02:41.970 --> 00:02:45.150 align:middle line:84%
Integration testing
tests components together

00:02:45.150 --> 00:02:46.080 align:middle line:90%
in a scenario.

00:02:46.080 --> 00:02:50.100 align:middle line:84%
And sometimes these can be
done prior to deployment

00:02:50.100 --> 00:02:51.810 align:middle line:84%
to an environment,
but very often, it

00:02:51.810 --> 00:02:55.120 align:middle line:84%
occurs after a deployment
to some type of environment.

00:02:55.120 --> 00:02:58.920 align:middle line:84%
I mean, these can take seconds
to minutes to run, generally.

00:02:58.920 --> 00:03:02.430 align:middle line:84%
Then you have user interface
testing, or automated UI

00:03:02.430 --> 00:03:03.160 align:middle line:90%
testing.

00:03:03.160 --> 00:03:04.860 align:middle line:84%
And here, we're
going to be driving

00:03:04.860 --> 00:03:07.650 align:middle line:84%
the tests through
the user interface,

00:03:07.650 --> 00:03:10.170 align:middle line:90%
and these take even longer.

00:03:10.170 --> 00:03:13.200 align:middle line:84%
Sometimes these can take
hours upon hours to run

00:03:13.200 --> 00:03:16.320 align:middle line:90%
or at least quite a few minutes.

00:03:16.320 --> 00:03:18.810 align:middle line:84%
Load and performance
testing is the next one.

00:03:18.810 --> 00:03:21.450 align:middle line:84%
And those, again,
should be automated

00:03:21.450 --> 00:03:26.750 align:middle line:84%
so you can compare piece after
piece, release after release,

00:03:26.750 --> 00:03:29.980 align:middle line:84%
and compare the performance
profiles so that you

00:03:29.980 --> 00:03:32.260 align:middle line:90%
can get continuous improvement.

00:03:32.260 --> 00:03:34.100 align:middle line:84%
But these take the
longest to run.

00:03:34.100 --> 00:03:37.450 align:middle line:84%
Now, you can hammer a site
or hammer your application

00:03:37.450 --> 00:03:40.240 align:middle line:84%
very aggressively for
a short amount of time,

00:03:40.240 --> 00:03:42.820 align:middle line:84%
but sometimes you
just need to run

00:03:42.820 --> 00:03:45.140 align:middle line:84%
for a fairly substantial
amount of time.

00:03:45.140 --> 00:03:48.580 align:middle line:84%
So these may tend to run
for quite a few minutes,

00:03:48.580 --> 00:03:52.300 align:middle line:84%
to hours, to sometimes
even potentially days.

00:03:52.300 --> 00:03:55.480 align:middle line:84%
The last types of test
listed here I've highlighted

00:03:55.480 --> 00:03:58.780 align:middle line:84%
in a different colour, and those
are the manual and exploratory

00:03:58.780 --> 00:03:59.500 align:middle line:90%
tests.

00:03:59.500 --> 00:04:02.590 align:middle line:84%
This is where you're going
to take your human intellect

00:04:02.590 --> 00:04:06.050 align:middle line:84%
and try to foil or
break the application.

00:04:06.050 --> 00:04:10.900 align:middle line:84%
And these traditionally are
very expensive tests, especially

00:04:10.900 --> 00:04:13.870 align:middle line:84%
in the context of a dev
test or a DevOps pipeline.

00:04:13.870 --> 00:04:17.140 align:middle line:84%
Because if you're building
things in an automated fashion

00:04:17.140 --> 00:04:20.680 align:middle line:84%
and you have to pull
in an individual user,

00:04:20.680 --> 00:04:23.530 align:middle line:84%
they can cost a lot of money
and take a lot of time.

00:04:23.530 --> 00:04:25.990 align:middle line:84%
And these are measured
on hours, to days,

00:04:25.990 --> 00:04:29.110 align:middle line:84%
to potentially even weeks
to run some of these tests

00:04:29.110 --> 00:04:30.490 align:middle line:90%
exhaustively.

00:04:30.490 --> 00:04:32.080 align:middle line:90%
So what do you do?

00:04:32.080 --> 00:04:36.003 align:middle line:84%
In your pipeline, we've
got the various components

00:04:36.003 --> 00:04:36.670 align:middle line:90%
in our pipeline.

00:04:36.670 --> 00:04:38.500 align:middle line:84%
And we're going to
start to build them out

00:04:38.500 --> 00:04:41.860 align:middle line:84%
with things like unit testing
right up front because they're

00:04:41.860 --> 00:04:43.300 align:middle line:90%
fast and easy to run.

00:04:43.300 --> 00:04:45.910 align:middle line:84%
And as we go through to things
like automated acceptance

00:04:45.910 --> 00:04:49.000 align:middle line:84%
testing, we're going to be
doing those with automated tools

00:04:49.000 --> 00:04:52.150 align:middle line:84%
that may not take
very long to run.

00:04:52.150 --> 00:04:54.430 align:middle line:84%
And as we get further
down, we might

00:04:54.430 --> 00:04:58.690 align:middle line:84%
start dropping in those humans
who can use their intelligence

00:04:58.690 --> 00:05:01.160 align:middle line:84%
and their will to try to
break the application.

00:05:01.160 --> 00:05:05.620 align:middle line:84%
And that's important too, but
often done right at the end

00:05:05.620 --> 00:05:09.370 align:middle line:84%
before you go to
production, if at all.

00:05:09.370 --> 00:05:12.220 align:middle line:84%
What are the benefits
of continuous testing?

00:05:12.220 --> 00:05:14.260 align:middle line:84%
They're myriad,
but here's a few.

00:05:14.260 --> 00:05:17.170 align:middle line:84%
Quality gates throughout the
pipeline-- different levels

00:05:17.170 --> 00:05:19.090 align:middle line:84%
of quality gates
throughout the pipeline,

00:05:19.090 --> 00:05:22.340 align:middle line:84%
and you can choose where
you put each quality gate.

00:05:22.340 --> 00:05:24.850 align:middle line:84%
And if the tests
fail, you can go back

00:05:24.850 --> 00:05:28.000 align:middle line:84%
and automatically fail
your entire deployment up

00:05:28.000 --> 00:05:28.780 align:middle line:90%
to that point.

00:05:28.780 --> 00:05:30.880 align:middle line:84%
And it comes back,
and we get context

00:05:30.880 --> 00:05:33.460 align:middle line:84%
back to the developers
and the operations team

00:05:33.460 --> 00:05:36.040 align:middle line:84%
that something's missing
or something's incorrect,

00:05:36.040 --> 00:05:37.600 align:middle line:90%
and we can go back and fix that.

00:05:37.600 --> 00:05:40.610 align:middle line:84%
It gives you increased
confidence as well.

00:05:40.610 --> 00:05:44.440 align:middle line:84%
You just get that really strong
understanding and confidence

00:05:44.440 --> 00:05:47.800 align:middle line:84%
that your code is going to make
it to production at a really

00:05:47.800 --> 00:05:49.730 align:middle line:90%
high level of quality.

00:05:49.730 --> 00:05:53.380 align:middle line:84%
Now, I want to add one
warning before we leave.

00:05:53.380 --> 00:05:56.350 align:middle line:90%
Don't tolerate broken tests.

00:05:56.350 --> 00:06:00.430 align:middle line:84%
If a test breaks and you've
got a red on your unit test

00:06:00.430 --> 00:06:04.570 align:middle line:84%
or a test continually breaks,
don't ignore that test.

00:06:04.570 --> 00:06:08.350 align:middle line:84%
At the very least, pull it
out of the testing pool.

00:06:08.350 --> 00:06:11.530 align:middle line:84%
What you don't want is for
anybody in your environment

00:06:11.530 --> 00:06:14.740 align:middle line:84%
to get this attitude of, eh,
just ignore that failure,

00:06:14.740 --> 00:06:17.020 align:middle line:84%
it happens all the time
anyway, and the code still

00:06:17.020 --> 00:06:22.000 align:middle line:84%
runs, because you'll then miss
the triggers that are informing

00:06:22.000 --> 00:06:24.230 align:middle line:90%
you when real problems exist.

00:06:24.230 --> 00:06:27.610 align:middle line:84%
So continuous testing-- whether
it be in the early automated

00:06:27.610 --> 00:06:31.090 align:middle line:84%
tests we run on unit tests, all
the way through to potentially

00:06:31.090 --> 00:06:33.940 align:middle line:84%
load tests and even
some manual tests--

00:06:33.940 --> 00:06:37.210 align:middle line:84%
are going to be critical
for your DevOps pipelines.

00:06:37.210 --> 00:06:43.510 align:middle line:90%


00:06:43.510 --> 00:06:46.860 align:middle line:90%
[MUSIC PLAYING]

00:06:46.860 --> 00:06:50.000 align:middle line:90%