BRIANNA BLASER: Well great, folks.
I think we'll go ahead and get started.
For those of you who I don't know,
my name's Brianna Blaser, with AccessCSforAll
at the University of Washington.
And as you may know, we work to help
ensure that K-12 computer science
education is accessible.
And we are always happy to work with Andreas Stefik
from the University of Nevada, Las Vegas.
And so today, Stefik is going to share with us
some of the work that he's been doing related to data science.
So I'm going to turn it over to Stefik
and let him take it from here.
ANDREAS STEFIK: All right.
Well, thank you, Brianna.
And it looks like I actually know quite a few people that
showed up.
Neil Brown, I see.
It's got to be like 2:00 AM there.
I don't even know.
So cool.
Well, thank you all for coming.
Brianna, can you confirm that my screen
is showing on the share screen?
BRIANNA BLASER: Yep.
Absolutely.
It's there.
ANDREAS STEFIK: OK.
Cool.
So, yeah.
Andy Stefik.
Everybody calls me Stefik.
I work in Vegas and I'm one of the PIs on AccessCSforAll.
We do a lot of accessibility work in high school.
If you don't know the kind of stuff,
the other PI on the project is Richard Ladner out
of University of Washington.
Also Sheryl Burgstahler as well.
So what we're going to talk about today
is some work that came about because of some changes that
were made to the--
in the United States, there is this entity
called the college board.
And they basically control classes in high school,
effectively.
And one of the units that they added
was related to data which caused the organization code.org
to create these data units.
And then the question became, well how exactly do you
handle this kind of data or data science type applications
in an accessible way?
Also this led to some other conversations
with Gary Ritchie, who is also on the call,
related to some work that are doing about accessibility
with data science in R as well.
So will this actually go?
There it is.
So I want to give just a little bit of fair warning
on this talk.
It's maybe not a good idea to give fair warning
and then watch all the people drop off,
immediately, at the beginning of a talk.
But there are a lot of talks that are sort
of like, what is accessibility?
And who has it?
This kind of stuff.
That's not this talk at all.
This is a technical talk.
This is a technical talk on how we
make things accessible in nitty, gritty technical detail.
So fair warning, I have tried to translate some of this stuff
into English.
But accessibility stuff, under the hood,
is actually kind of complicated.
It just is the nature of the beast.
It just is how it is.
So this talk is about what data science is and it's challenges
and how accessibility works and then how we evaluate it
in the context of graphics or anything data science related.
Now, the reason I'm talking about some
of the technical stuff is because even if you're just
evaluating your team's accessibility of a product
that you're making, believe it or not,
a lot of the words and symbols and phrases
that exist in code are actually the things
that the accessibility tools that track things,
they use the same terminology.
And it's wacky.
So I want to make that correspondence
between those things clear so that when
you do, even an accessibility check,
you know what you're looking at and what
you're looking for when you're doing this kind of work.
So funny thing.
So data science, if you're not familiar--
if you're not a data scientist--
I know some of you are.
I know Gary is.
And I know some of you have written some really cool data
science tools.
I know Neil Brown has these really cool spreadsheet
type tools and stuff like that that I
think are super impressive.
But data science is a weird thing.
One of the things we often say in publications,
or what we've been saying lately,
is it's kind of the wild, wild west.
And the metaphor isn't that these people are all gun-totin',
and I wouldn't really know that.
But the metaphor is that it's not the most organized space
of thinking on planet Earth.
A lot of the libraries just use weird naming conventions,
they're not consistent anywhere, they don't match textbooks.
It's just sort of a strange discipline, broadly.
But the problem is, though, the fact
that it's the wild, wild west is kind of irrelevant because it's
also really important.
Empirical data and statistics are used pretty
much in every discipline now.
It's not really controversial to use evidence anymore,
at least not since the Dark Ages, I guess.
Funding agencies nowadays often require evidence.
This is not shocking.
We're in more a scientific community
than we have been historically.
And data science pays pretty well.
So people want to do this kind of work because it's lucrative.
You can make money being a data scientist.
And then it's become so common that even in K-12,
we're observing curriculum that may not be called data science.
For example, code.org's unit 9, they
don't call it a data science unit,
but there's clearly a relationship
between data science and the kinds of content
that students are doing.
It's sort of an early intro to data science in a sense, right?
Depending on how you look at it.
So then the question becomes then, well, what is it exactly?
What is data science?
And, you know, there is no universal definition,
I would say.
But there are groups that are kind of
trying to define this in some formal sense or the other.
And I'll mention, too, there's a new report out of Harvard where
they tried to kind of talk about this.
And the one that I'm probably more familiar with,
just because I'm sort of loosely connected to that committee,
is the ACM Data Science Task Force.
And they define it as having 11 knowledge areas.
And of course, keep in mind, this
is a bit subjective, right?
Exactly whether these are the right 11
is unclear, but not the point.
They include things like data analysis, presentation, or AI,
or big data, or computing fundamentals, acquisition,
management, and governance, data mining, data privacy,
machine learning, professionalism, and stuff
like that.
If you're not familiar with what some of the thinking
is, at least in academia, on what data science is,
you can actually read the draft report of the ACM Data Science
Task Force at dstf.acm.org.
That's draft two.
I think draft three or four will end up being final,
although I'm not positive on the of the exact stuff there.
So what do people do with data science?
Well, this is also difficult to answer.
There are some interesting surveys on it,
and there's some really good reports, too,
that have been coming out over the last few years
on the kinds of things that data scientists do.
But one example of this might be the Stack Overflow survey
from a few years ago where they looked
at just what percent of developers
do stuff like statistics.
And it turns out to be around 11%-ish, 11.3,
according to their survey.
I read this is Stack Overflow, so you can decide for yourself
whether that sample is actually generalizable.
But nonetheless, it's one data point that we can think about.
And of that, I think it's ballpark nearly half ended up
using Python, R, or Matlab.
And there's surprisingly little evidence on the sort
of tools on language designs, on the accessibility,
really, on almost anything in relation
to how people are doing data science.
It's only been in the last few years
that you've been really seeing a lot of publications come
out that contain data on the kinds of things
that data scientists do.
There's a few of them.
We've written some of them, but we're not alone.
Austin Henley has a really good paper at out of CHI
And some other groups, as well, have done some cool work here.
So one of the things that we did after talking
to Gary Ritchie and his team at our studio over the summer
was we did a quick analysis of all the tools that we find.
And we went in and just looked at what is accessible
and what is not across these.
In part, just because I was curious, but also because
it's not really that difficult to download them.
And say, does it turn on with a screen reader
and what other problems does it have?
Things like that.
And so I'm not going to read this whole list.
You can reference to the slides if you want to.
But I want to give you a sample of some
of these verbally, just in case there's viewers
that can't see the screen.
These are tools like RStudio, CODAP, iNZight, DataDesk,
Fathom, Jupyter Notebooks, IntelliJ, SpyderJ, Thony, Atom,
JMP, MATLAP, RapidMiner, SAS, SPSS, Statview, TinkerPlots.
And there's others, as well.
There's groups from Tableau, there's
groups out of SAS that do what's called the graphics
accelerator, which is a really cool tool related to especially
sonification, but also some speech, as well,
in terms of sonifying data.
There's also whole communities that
have been thinking about sonification
for a long time with the ICAD community,
the International Conference on Auditory Display
and stuff like that.
I'm not going to go through all that research literature.
There is some of it out there.
But I just want to make it clear that there is a list of these,
and when you look at the accessibility
across these tools, there is a lot
of accessibility challenges.
And when I say challenges, I don't
mean people are doing a good or bad job.
That's not really my point.
My point is that in some cases, groups just
didn't have support for certain things.
And in other cases, it's just really hard
to figure out what it should be to be accessible.
So there's a lot of accessibility challenges
in data science.
But don't take my word for it.
There's this wonderful fellow named Jonathan Godfrey,
who I don't really know.
I've emailed back and forth with him a time or two.
And he's written a few papers on this topic.
And they're really neat.
If you don't know them, they're worth taking a look at.
One of them is this paper that he
wrote in 2013 called Statistical Software from a Blind Person's
Perspective.
In that particular case, I think he
was using R, one of the variations,
I don't think it was RStudio at the time.
I think it was one of the other tools.
But that's not really the point.
The point was that he was using one of the one of the R
just default packages.
And Jonathan has a bunch of newer work, as well, where
he talks about how do you do accessible graphing
and how do you do accessible this and that.
And they're really fascinating and give
a lot of really good ideas about what
it might mean for data science to become more accessible.
And they really, at their core, I think, at their core--
and, you know, he might totally disagree with me.
At their core, they really come down
to having more structure inside of these kind of tools.
And I can get to that in a bit.
The key lesson, I think, is really
that there's still pretty major accessibility barriers
in data science.
Just massive, massive barriers.
So the next thing I want to do is I actually have a pop quiz.
And I know you'll all love that.
That's exactly what you want in a webinar.
So if you don't know how to do it,
go ahead and pull up in the screen.
There's a chat window.
If you're a screen reader user, you
press Alt h to get to the chat window.
And if you're not, you can either press that hotkey,
or you can just go.
And what I'm going to do is I'm going to have three sides.
And what I'm going to do is I'm going to give everybody
about 30 seconds to actually discuss
what they imagine the accessibility barriers are
to a particular thing.
And so in the first slide I have on the screen,
I'm going to read the text in case you can't see it,
some code.
And I want the accessibility barriers
to be discussed for just the code, not the tools around it,
not how you interact with it or anything like that.
Just the words and symbols that are on the screen.
Are there accessibility barriers here?
So the code is library(Lock5Data),
data("SandwichAnts") m1 less than -aov(Ants~Fillin
g,data=SandwichAnts), summary(m1).
OK?
Now you all have 30 seconds to come up with any ideas
that you think might be related to accessibility and toss them
in the chat.
I'm going to be reading that.
So give it a shot.
I know it's a little weird, but it's
a good way to get interaction.
Screen readers can't read that in paragraph.
Aov is not pronounceable.
Leaving spaces versus leaving tabs.
Numerals versus spelling numbers out.
How do you pronounce tilde?
That's good.
You got 10 seconds.
Summarym1 has no spaces.
Tilde's not used often.
OK, well we have to stop.
If I'm new to the field, what does this even mean?
Aov?
Yeah, exactly.
Oh, that's Jenna.
Hi, Jenna.
OK.
So let's break that down.
Now there is actually quite a bit
of evidence on words and symbols in the academic literature
related to programming languages.
And I'll just briefly mention what some of these are.
Oftentimes, people might assume that a particular word has
a particular meaning to humans, but the problem is humans vary.
So I'll give you an example.
Less than dash might look visually like an arrow,
but actually, even to novices, the less than dash
does not mean assigned to.
It does not mean that.
To novices, it's been formally tested in studies, right?
We did one of them.
And in addition, a lot of the words that
are used in programming languages--
this happens to be R, but it's relevant.
Matlab and Python are very similar in terms
of their choices.
Terms like a aov are shortened acronyms
that don't match either the academic literature,
don't match textbooks, and don't match Matlab and Python,
either.
And so these terms actually don't mean anything to people.
When you have certain kinds of characters like tilde,
tilde has a particular semantic in R
and its meaning is kind of wacky.
And I have to admit, I only mostly understand it, I think.
I know how to use it and when, but exactly what
the deeper language meaning is is maybe
a little bit elusive to me.
I don't know.
But the point is, to a novice, the tilde
doesn't mean what you probably think it means,
I would imagine.
Stuff like that.
Oh, there's also Good Data by Neil Brown and Brett Becker
and some of those kind of folks on parentheses matching being
a particular issue for novices.
This isn't just accessibility, but it's for everybody.
But it is an issue that shows up in their data quite a bit.
So those are a couple of things just
related to the text, just related to accessibility.
It's not an indictment of any particular group.
It's just observations based on facts
from the literature about the kinds of issues
that people have.
So let's go to the next one.
So that's program input.
So now let's go to program output.
And for this one, I actually have
to change what I'm sharing on the screen for a second
because I want you to observe it in a different kind of way.
So I'm going to stop sharing and I'm going to pull up this code.
And what I'm going to be doing is
I'm going to actually pull this up in a screen reader.
And I'm going to share a screen as a whole
but I'm going to do it with shared sound.
So I'm going to pull up Notepad and I'm going to turn on NVDA.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: And I've intentionally
made NVDA ridiculously slow to try
to make it easier to understand if you're not a screen reader
user.
So I'm going to dump in some code.
I'm going to dump it to the top.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: OK?
Now the first thing is if I'm navigating with a screen
reader, I'm not going to give you a screen reader tutorial.
But I'm going to kind of navigate through it
a little bit to give you an idea of how
you would interact with this.
And then I'm going to give you another 30
seconds to try to derive for yourself mentally how you would
do this with a screen reader and what
the accessibility problems are.
So if I want to know, for example, what the value of f
is in this ANOVA that I've put in, what I have to do
is I can either go press up and down.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: Mentally track the columns
and then press down.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: OK?
Or I can go in and I can navigate
by things like columns.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: If I do this, I then have to mentally line up
that mean SQ and sum SQ are actually columns,
but they're two screen reader calls because of the press Ctrl
right twice.
And then I have to go down--
NVDA: [INAUDIBLE]
ANDREAS STEFIK: --go to the left,
and then I have to line up those columns
with the imagined columns by lining them up a different way
mentally at the bottom.
NVDA: [INAUDIBLE] 1561.
785.627.
ANDREAS STEFIK: And if I do that mental line up,
I can track in my head which columns match to which.
And then all of the sudden, I have
text from R translated into something
accessible through a screen reader.
OK.
That's a very, very brief tutorial
about how you might get some output--
NVDA: [INAUDIBLE]
ANDREAS STEFIK: --from something like R. So 30 seconds again.
Go ahead and chat any other issues
you think with having a screen reader with specifically
the unstructured output that is often
given from these environments.
30 seconds.
Less than, dashes, and arrow but would be read
as two separate characters.
Yep, that's definitely true.
That's input, but I guess that's from the last one.
I see people typing, or some.
Everything on the screen is too hard to read.
Oh.
I'll try to make that bigger.
Sorry about that.
If you aren't used to the field, arranging this in your head
is not intuitive.
Definitely.
Some of the spaces within columns,
space characters, some of those column separators tab.
Oh, yeah.
Actually, that's probably true.
None of the visual layout clues are automatically
available to screen reader users.
Yep.
High cognitive load, almost certainly.
Mapping of rows and columns is difficult. Definitely true.
It's not on a table.
Yeah.
That's really the simplest answer to this, actually.
Difficult to go down within a column.
100% true.
Empty cells cause off by one errors.
Totally.
Yeah.
All these things sound right to me.
So why don't we stop there.
OK now there's one more of these exercises that I want to do.
And I'm going to share my screen again.
And I'll make it bigger so that everybody can see.
And I'll-- oops, wait.
I need to Present Mode.
And-- sorry, just one second.
OK.
And when I do this, I have now on the screen one more,
in this case, it's a graph.
And this graph actually comes from code.org's data unit 9,
although that doesn't matter.
It's really just a graph.
And I'll describe it in English so
that people that can't see it can know what's there.
Basically, this is a graph that looks
kind of like a longitudinal graph where
it's tracking certain blips in the data over time.
It's got dates on the bottom.
February 15th, 20, dot, dot, dot.
And then on the right on the bottom,
it's September 1st, 2019.
On the left axis, there's 100, 75, 50, and 25.
And then it's basically a line chart
that has a bunch of little wobbling in it.
And then it has these occasional blips.
And there's five of them that seem to go at a relatively
periodic pattern, OK?
And there's a couple of things to notice here
that I'll mention.
But again, I want to give you 30 seconds
to figure out exactly what are the challenges here
with making this accessible and what makes this--
what are the challenges here with this kind of stuff?
So 30 seconds.
Go.
If it's just an image, no better data for screen readers.
Yep.
Glance exposure is not something that can be effectively done
by a text-voice descriptions.
Yep.
People that have never seen a line chart
won't understand the structure.
Yep.
In describing it, you put your own interpretation on it.
Yeah, exactly.
Even if you have a descript-- yeah.
The peaks and shapes are significant.
Yep.
You have to step--
oh, Gary has to step away for a few minutes.
Not an accessibility issue, Gary, but I'll accept it.
Need some way to interactively explore graphs
and read off chosen points.
Yeah.
All these things are currently true,
so why don't we stop there.
OK.
And I'll add just a few others.
One of the challenges with accessibility,
oftentimes, is we have this sort of knee-jerk reaction
to say, well, let's just put an alternate description
in our website or something.
And an alternate description is absolutely better than nothing.
I mean, not having any information at all
is clearly bad.
But the problem is the alternate description
makes the presumption that the person that
wrote the description actually is telling you
the information that you want as a user,
and that may not be the case, right?
You might want all sorts of information about this graph.
You might be looking for what's the scale,
and that wasn't in the alternate description.
You might be looking for when are the blips?
What is the periodic pattern of the blips?
Or any other number of other things
depending on what field of study you're in.
And I'll tell you, having collaborated
with groups in other fields--
one of our projects is in making astronomy more accessible.
The kinds of information that they look for in graphs
is very different than when I run rat-controlled trials
on programming languages or other topics.
It's just the information we're looking for on these charts
is very different.
No auditory description would be good enough because there's
other things we're looking for.
So I kind of wrote these up as a roundup in the slides.
And I'll just briefly touch on them.
So library designs.
And basically every statistical language that I've looked at
have poorly thought out acronyms.
Functions named after mathematicians.
Really shouldn't be called Crucible Wallace.
It should have a meaningful name, right?
Single letter label names.
F-test, t-test, p-value, b-distribution, d-distribution,
gamma distribution, all this kind of stuff.
They make no sense.
Names that lack semantic meaning,
like anything with the words gg in front of it,
regardless of the library that is.
The tilde operator.
No global organization system.
Oftentimes, these things have been
built by giant communities.
And that makes sense.
That's how you get stuff built. But they
lack any kind of like, hey, does your name make any sense?
Hey, what's the evidence for your naming convention?
Or a global organization is just totally missing
in data science, I think.
And that's not an easy problem to solve
nor is it an indictment.
It's not like we all go and say, aha!
We'll wave the magic wand and then
we'll all be super organized.
It doesn't work that way back in reality.
But nonetheless, it is true that it lacks a global organization
and naming system, right?
So also it's not just the languages, though.
Programming languages, in many of these fields,
they actually lack accessibility support at the language level.
And these have very specific libraries.
For example, in Windows, it's called UIA.
I'll talk about that later.
In Apple, it's called NSAccessibility.
In Android, they have a Talkback API.
And on the web, they often call it
ARIA related to this thing called WCAG,
but actually, those just connect to UIA and NSAccessibility,
so it's complicated.
But when this support is lacking,
that means any tool that you build
on top of those programming languages
is automatically not accessible.
Just out of the gate, there's nothing
you can do about it unless you add those libraries.
And I'll talk more about why that is and what that is.
But a couple more in terms of accessibility problems.
Oftentimes, there's input problems
like no or poor keyboard support, no virtual keyboard
support.
This actually sounds weird.
A virtual keyboard is not your physical touching thing.
Virtual keyboards actually work a little bit different
and it's really easy to screw up virtual keyboard support
in accessibility technologies.
I've screwed it up myself before.
And then there's lots and lots of issues related
to graphics and structure.
No resizing of text or images, no alternate descriptions,
graphs render without explorable structure,
unformatted raw output with really silly naming
conventions, inaccessible PDFs that get
generated, videos that lack closed captioning,
all sorts of stuff like that.
The point isn't that this is a total list of all
the accessibility issues, I'm just
trying to give you a flavor of the types of things
that I see when I do accessibility analysis on tops
of all the tools that I mentioned previously.
These are the kinds of things that we
observe on a factual basis.
OK, cool.
So I just told you about the billion hours of work
that would be needed to fix all these issues
across these tools, but how do you even do it?
Because it turns out that accessibility and testing
for it is kind of complicated and it's not super easy.
And in fact, because the API differ across every platform,
I really can't, in one hour, describe how
this works for every platform.
It's just not possible.
These platforms are incompatible and accessibility
is 0% compatible with UI Automation on Windows,
and same with the Talkback API and similar stuff, as well.
So what I'm going to do is I'm just
going to talk a little bit more today
about one of these platforms to give you a flavor for how
they work across each.
And then that will hopefully give you
a sense of how you do testing, at least on one platform.
And it's different on the others.
So what I'm going to focus on is called UI Automation
on specifically Windows.
And I'm doing that one in part just because I
know it the best, but also because it's really mature,
and so you can do accessibility black magic with it if you
know what you're doing, which is neat.
So what is UI Automation?
Well, it's a low level, C or C++,
way to interact with the operating system
for specifically accessibility.
It's low level.
It's really, really gross, but it is functional.
And if you want any programming language at all
to have accessibility support on Windows specifically,
especially when Windows 10 it's kind of the only game in town.
There are other historical techniques
that have been used for accessibility
on these platforms, but UIA is the solution today
for doing accessibility work.
And UIA has two things you have to know about it.
It has clients and it has providers, OK?
And this is slightly tricky, but the core lesson
is products made in programming languages that
lack this mapping on these operating systems
are automatically not accessible.
You just-- out of the gate, they won't work,
or they'll work very poorly for people with disabilities
on a lot of fronts.
And I'll kind of explain why that is.
So first, let's talk about clients.
When UIA clients access information
from the operating system, it wouldn't make sense for them
to have to go to every single application that's
ever written on Earth and say, hey, what do you provide?
What do you provide?
That would be too difficult. So what they do instead
is they connect down to UIA and ask any application that
is provided the correct mappings, what do they support,
right?
And so these applications vary a lot.
So the classic one is a screen reader that--
people know about that one.
But people often don't realize when Microsoft
made accessibility support, they didn't
make screen reader support and then virtual keyboard support
and then magnifier support and stuff like that.
They all run through the same architecture
because they don't want to go crazy, right?
It's just simpler to do it that way.
So when you break something for, let's say, a screen reader, you
might actually break it across a whole range
of accessibility products, right?
Virtual keyboards, sip and puff readers,
which are on the screen, screen readers, Braille devices,
magnifiers, all sorts of stuff.
They all use similar technologies under the hood
through UIA.
So that's the idea.
And they look at clients, look at information
provided through UIA through event structures and structure.
So if it's a screen reader, it connects UIA,
it gets its structure, and it listens to its events,
and then it responds accordingly, right?
So lesson two.
Clients consume information across all disability types
with the same architecture.
And on Windows, that's called UIA.
On Mac, it's NSAccessibility.
And UIA also has providers.
And providers, that's what individual applications have
to write in order for their content to be accessible.
So if you're making, hypothetically, RStudio, you
need to have a provider that sends information
to the system for your application.
And that may or may not be easy to do,
depending upon the programming language.
It's not flip a switch, right?
It's not simple.
So what providers do, they provide three things.
One is control patterns, one is properties, and one is events.
Those three are critically, critically important.
I'm going to show you a couple of them.
So sometimes when people email, they
ask why is the Quorum programming
language accessible?
It's because we have a really fancy provider.
That's really what it comes down to.
Quorum has a really fancy provider
that can do stuff for all sorts of kinds of applications,
and that's it.
That's really what it comes down to.
OK, lesson 3.
If you don't implement a provider or one is inside
of the programming language you're using
or it's a bad provider, as happens,
Java has kind of really terrible provider,
your application, anything built on top of that,
is not going to be accessible or it's
going to be poorly accessible.
So let's look at a couple of these.
And again, I'm going to share a screen
in a different way for this so you can actually
see some of these.
They're not that complicated, but I
have to share them in a different way.
So first what I'm going to do is I'm going to show you a low
level C++ provider for a button.
I'm showing you a button because it's by far the simplest thing.
And you'll notice that this is low level C++.
And if you're already thinking, oh, dear God, Stefik,
I'm going to vomit, then you're right.
That's the correct answer and you should know that.
But it's the abstraction that we have,
so you just kind of have to deal with it.
It is the reality.
And also the problem UIA is solving
is actually very complex.
And so I have to give Microsoft props
that even though the abstraction is kind of gross-looking,
it does work, and that's important, right?
It does actually work across these products.
OK, now I'm not going to read this code to you,
but I do want to point out a couple of things.
And they'll come back later, which
is why I bothered to show you some source code.
Inside of this funny thing, I have a thing
called a button provider.
And inside of it, it implements this thing called an IInvoke
provider.
And you might think, dude, that's super low level
and detailed.
Why do I need to know that?
And the reason is because when we evaluate
accessibility a little bit later,
these are the same names that show up
in the accessibility tools.
So if you don't know the low level names,
you won't understand what it's telling you
when you get back accessibility information from the operating
system.
Unfortunately, they're linked and they're not
translated into English.
It just is what it is.
So there's this thing called IInvoke provider that's
basically telling the operating system, hey, I have a thing
and I'm declaring to you that it's a button.
What is it on screen?
Doesn't matter.
Spinning dragon?
Cool.
It's a spinning dragon button.
Right on.
Is it a fire-breathing monkey?
Of course it is.
Fine.
It's a fire-breathing monkey, but it can be invoked.
Doesn't matter.
But that's the cool thing.
You can declare.
And this is how you can make things like graphics
accessible.
Hopefully that's a hint, right?
But there's some other stuff.
One is there's this thing called, IsPatternSupported.
This is kind of a thing that we do in the Quorum architecture.
And I'll get to what that is once we
get to the implementation.
And then there's this thing that says getControlType which
is really, really important because it has to do with how
things interact with your tool.
And then finally, there's this thing, and I love the name.
It makes perfect sense, obviously, it's very intuitive,
called IFACEMETHODIMP, right?
Obviously, I always think of a garden gnome
that's, like, sitting around--
I don't know.
This is saying this is an interface method
and it's an implementation of it.
It's a convention in Windows programming.
I'm sorry.
And it says invoke.
This is saying, if you have that spinning, fire-breathing monkey
on the screen, and the AT technology, no matter
what kind it is, wants to tell it to invoke it,
to click the button without actually
pressing a key or something with that, they can do that.
And this then triggers an event that sends it up
to your system.
It's sort of like--
people with different kinds of disabilities
interact with computers in all sorts of ways.
And they can declare to the operating system
that they want to invoke a button,
even though they may not be physically pressing the key.
They might have a sip and puff, they
might have any number of other technologies
that they're using, and you can't
know as the application developer what
that is because it's however they choose.
OK.
So then there's an implementation.
And I'll show you this really quick.
It's really not that much code.
It's relatively small, but it's esoteric.
And that's the weird part.
So I have this IsPatternSupported,
and you'll notice that there's this funny thing that
says, if the pattern ID provided by the operating system
matches UIA invoke pattern ID, that it returns true.
And this sounds bonkers, but what
it's telling you is it's saying, I am a button.
That's what this is saying.
And when we pull up in a moment the accessibility checkers,
you will literally see a thing that says, hey,
this is a button.
And it will tell you this pattern is provided.
That's why I'm showing you this.
And then in addition, there's this thing
that says the control type, and this is just
a number under the hood, is a UIA button control type ID.
This is critically important.
And your accessibility will be completely broken
if this is wrong because this tells something like a screen
reader or other device what kind of thing it is, which tells it
how it reads it to the user.
So it says, oh, this is a button?
Tell the user this is a button verbally, or whatever, right?
And then finally, this invoke NR system, this
won't mean anything to anybody, but it just
passes an invoke event back to us.
OK.
Now I want to show you one more of these providers,
just so we can get a slightly hard one.
And so let's pull up a checkbox.
And I'll go a little bit faster for this one.
In a checkbox, the important thing is instead
of an IInvokeProvider, I've got an IToggleProvider
because it toggles things on and off.
And then I've still got the same two methods,
whether it supports a pattern or not,
And then I've got a function to toggle and a function
to get toggle state.
And I'll show you why this matters
in just a second, because it tends to do with how
it sends information around.
So I'm going to pull up the implementation here.
And in this particular case, my pattern
is a UIA toggle pattern ID.
That tells the operating system how it
can be interacted with, right?
What commands can be sent back and forth.
And then I've got a UIA checkbox control type ID.
This tells the operating system, hey,
this spinning, fire-breathing monkey can
be flipped on and off, right?
Again, the graphical representation
is entirely irrelevant to the operating system.
All it cares about is what kind of thing
am I declaring that this is.
I could be completely lying.
And in fact, in Quorum, we repeatedly and consistently lie
to the operating system all the time.
That's just normal.
OK.
The last thing I want to mention here
is when I actually have a toggle,
you have to do some weird black magic.
And I mention these things because there
are certain things in UIA that are kind of broken,
and you should just be aware that when you're
getting inconsistent screen reader behavior,
this is why it happens.
Notably, screen readers pass these things
around all the time called variance,
and they're truly disgusting and I
won't go into the details of what they are.
But the point is you can't just set a variable
and pass it around.
In screen readerland, you're doing
interprocess communication because your application is
telling the operating system that something's
available outside of your process,
and then another application outside of your process,
too, is retrieving it between there.
So the only way to do that in Windows is
do these funny variants and then send them around
through this thing called a UIA raise automation
property changed event.
Effectively, what this is telling the operating system
is that my checkbox has changed its value.
My spinning, fire-breathing monkey is now on or off.
And then a screen reader or other device
can choose whether or not to tell
the user about that effect.
Last thing I'll mention, there's this funny line here
that says if UIA clients are listening.
In theory, what this function is supposed to do--
this is a Microsoft function.
It's supposed to tell you if the screen reader is active
and then respond differently if it's there or not.
But actually, if you don't know, this doesn't actually work.
It's just broken in Windows, as far as I know.
I've confirmed this with Microsoft developers
that as far as they can tell, it doesn't work, either.
But as a convention, when you're writing accessibility code,
you always want these little monikers in there
just in case, because if they ever fix it,
I'm scared as to what would happen if you
don't have these in place.
So unfortunately, this is the reality
of accessibility coding.
Sometimes you have to be defensive in ways
that you wouldn't know make any sense at all, right?
This is in the Microsoft documentation,
but it doesn't work, yet you still
have to do it because reasons.
Welcome to accessibility technologies.
OK.
That's the idea of that particular type of thing.
So now I'm going to switch back my shared
screen to just the presentation and keep going here.
OK, so in this particular case, the last--
uh, put this down.
The last thing is for certain kinds of graphics,
and this is where we get back to data science,
graphics themselves aren't inherently accessible or not
accessible.
Any kind of graphics can be made accessible.
But inside of operating systems, they
don't always have providers for the kinds
of things you would want.
So for example, in the Quorum Studio product
that we have that we use at schools
for kids with disabilities, there
is no such thing as an infinitely sized grid
in a provider.
That doesn't exist.
So how do you do Unity 3D style things?
It turns out you can, you just have
to be kind of clever about how you manage your providers.
There's also no control type for a scene designer.
That's not a thing, Unity-style scene designers.
But there are ways to do it.
And I'm not going to go into the details about how,
but making providers for content this graphical and accessible
is totally possible.
And we do it today and it works just fine.
And it turns out if you solve problems
like this, which we call the accessible graphics problem,
you can do data science, block programming, 3D modeling,
shaders.
Doesn't matter if you're blind.
Totally not a problem at all.
It's just fine.
And we solved that only recently, fully, in Quorum 8.
So you can look at the implementations
under [? BST, ?] so you can use it, as well.
Point is solving the accessible graphics problem solves issues
across a lot of fields, including data science,
and you do it through these operating system hooks.
That's the trick that you have to do
to make all this stuff work.
OK.
So now let's move briefly to testing for accessibility.
And then I'm going to show you some graphs that
were designed through these successful graphics things
so you can see their properties and see what
they do in one particular AT device, which is a screen reader.
So a couple of tools to be aware of when
you're testing for accessibility, one
is called Inspect.
And I've put links to it in the slides.
And another is called Accessibility Insights.
If you're aware of it, Inspect is a little bit older.
It looks really clunky, but it also works pretty well.
Accessibility Insights exists and it's new
and it looks really pretty, but as far as we can tell,
we never really use it because it doesn't really
work all the time and it's kind of funky.
I don't know.
It might just be us using it wrong,
but we always use Inspect because of that reason.
And these tools give you a lot of information
about accessibility.
And so I want to highlight on the screen,
there's a little region where it says
is annotation pattern available?
IsDragPatternAvailable?
Is drop target available?
IsMultipleViewPatternAvailable?
All these patterns.
These are the literal patterns that come in from the code
that I showed you earlier.
And in addition, inside of this thing,
there's also this thing that says control type.
And it says UIA menu item control type ID.
This is the literal thing provided by the provider
to the operating system, and any accessibility technology
can choose to read this or not.
And the screen readers vary in what they choose to do.
In other words, if you get your accessibility
100% right on the native operating system level,
you still may not get support inside of a screen reader.
And that's not necessarily wrong because different screen
readers choose to interpret these events in different ways,
and you kind of just have to know.
As well, be aware, the screen reader JAWS
is at least a few years behind on their support for UIA.
And so as such, screen readers like JAWS
actually give inconsistent information
compared to screen readers like NVDA or even Windows Narrator,
actually, is a little bit more correct
on this particular front of interpretation.
So just be aware.
If you're getting trouble with one screen reader
and it works in another, there's sometimes really good reasons
for that, and you should ask an expert because sometimes they
would know.
OK.
So let's take a quick demo.
And I'm going to go back to sharing
the screen up above so you can kind of see
how this works in practice.
And I'm going to pull up Quorum Studio, the latest
version, which is 2.2.
And inside of it, I'm also going to pull up the Inspect
tool first, without a screen reader active,
so you can see what kind of information is provided.
And then I'll show you some stuff with some graphs.
OK.
So hopefully you can see my screen again.
And I'm going to pull up this tool, inspect.
And I'm going to just click quorum studio
and I want you to see what's in there.
So inside of this, you'll notice that visually on the screen,
believe it or not, Quorum studio is actually
not native controls.
It looks like it has tabs and it looks like it has little icons
and it looks like it has trees and affordances and all
this kind of stuff.
It's not.
It's raw OpenGL that is rendered on the graphics chip.
It has nothing to do with native controls whatsoever.
It's a big total lie to the operating
system as to what is happening.
Now why is that?
That's because we want to allow 3D things to be accessible.
We want graphics to be accessible.
We want graphs to be accessible and we really can't with native
controls because they don't work that way.
So if you look inside of this, you'll
notice that what we've done then is
we've set a bunch of providers.
So right now, it looks like it has the projects tree,
and that is a UIA tree control type ID.
So a screen reader or other AT device
would think that that's just a tree.
They wouldn't know that there's anything different.
It's just a normal tree.
I can use it in the normal way.
Same with if I have a text box.
In this particular case, that's a UIA document control type ID.
And you can notice that when I click the focus
inside of Quorum studio, the operating system
is informed that this particular type of thing has changed.
And then inside of the providers, if you don't know,
implementing a text provider's easily the hardest one.
It's very, very difficult to get it exactly right.
By having the text pattern available and the volume
pattern available, all of a sudden,
you get Braille support for free.
And you wouldn't even know that unless you just
magically know that this is how this works internally.
And so this is one reason why a tool like Quorum studio
can connect to Braille devices, because we just
know that these text patterns are related
to the way Braille support is done because the Braille
support connects to UIA just like a screen reader would
or any other device.
OK, so let's pull up a chart inside of this.
So I'm going to pull up a data science library.
I've been working on a statistics library, a data
science library, for, specifically, the Quorum
language.
And if I pull this up and I look at the UIA,
I have on screen, a small--
I mean, I just did it that way.
A small little chart.
And when I pull this up and that gets the focus,
you can notice that in UIA, there's actually
a lot of junk that gets put in.
And this looks like it's giving a lot of information.
It's putting in individual points and all sorts of stuff.
The structure of this tree, because of the way
we're doing it internally for events like this,
is kind of funny.
So just kind of ignore the fact that it doesn't look tree-like.
But from a user's perspective, it
doesn't matter at all, because we've decided, we've defined,
how they interact with these charts.
And in addition, you'll notice that inside of here,
there's this funny thing called a UIA custom control type ID.
This is a flag to the operating system that says, hey,
we know what we're doing.
We will take care of it for you.
Just chill.
And so when you do that, you have
to be very careful about what you're telling the operating
system at what times because you don't get things like selection
like you would in a tree, which are hooks to specifically
screen readers.
Instead, you have to manipulate the system
through focus events, not through selection events, which
is kind of an esoteric detail, but it's important.
So when I do this discreetly, you'll see what I mean.
So I'm going to pop this up NVDA.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: And I apologize that the scaling
is a little bit funny on here.
We're having a couple issues with scaling.
This is all brand new technology.
We literally deployed it Tuesday,
so it's not like some long-term thing
that we've had for a long time.
You'll notice that if I have a screen reader on
and I press the arrow keys, this looks like a chart,
really, to a sighted user.
You wouldn't know that this is not a chart.
It just displays graphics like anything else.
But If I press the arrow keys--
NVDA: [INAUDIBLE] from 0.0 to 1,120.0.
ANDREAS STEFIK: I can interact with the chart
in any kind of meaningful way.
This is exactly like what Jonathan Godfrey put
in one of his publications, except instead
of just providing structure randomly,
it provides operating system hooks so that the user would
just not know that this just magically is
accessible out of the box.
They don't even have to do any extra work.
In addition to that, if I press the up and down keys--
NVDA: [INAUDIBLE]
ANDREAS STEFIK: I can interact with things.
NVDA: 1,120.0505.
ANDREAS STEFIK: I can interact with pieces
of the chart in ways that we thought
made sense for accessibility.
Now of course, we haven't done user vetting with users yet
to know exactly which pieces people will like and which
pieces they won't.
But the point is that by having this UIA structure
and defining a set of keys, now we can go do testing with users
and see which things they want changed and stuff like that.
So if I get to the actual plots--
NVDA: FS scale is [INAUDIBLE].
Scatter plot 25 points.
VP 0.0 to 11,460.0.
[INAUDIBLE] to 1,120.0.
[INAUDIBLE]
ANDREAS STEFIK: This gets a little bit tricky
because if you had something like a scatter plot,
hypothetically, this could be thousands or millions of points
that you might have on the screen.
So the question is, how do you deal with that?
There's a number of ways.
One is, I think I saw Sean Mealin on the call earlier.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: But in any case, I
know with stuff like the graphics accelerator,
they try to give you the gestalt through sonification.
That's one way that you can try to do it.
You might also want to get at the individual points.
So one of the ideas we had here was
we can actually pretend that a scatter
plot is, in fact, a quad tree.
If you're not a graphics person, this
is just a way of breaking up the structure in a certain way.
And it looks kind of like this on screen.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: Oops.
Does not look like that at all.
NVDA: FS scale.
Scatter plot 20 [INAUDIBLE] quadrant [INAUDIBLE]
to 5,730.0.
FS 560.0 to 1,120.0.
ANDREAS STEFIK: I can break up my thing into quadrants
and give it information about what is in there.
In this particular case, it's saying
there's no points in this region,
here's the scale for this particular region,
so you kind of know where you're at.
My hunch is when we do the user testing,
they'll want us to shorten these queues a little bit
to make them a little easier to understand,
and they'll probably want us to change, maybe remove
some names, or do some adjustment like that.
But we'll find that out when we start doing testing.
NVDA: Top right quadrant 11 points.
VP 5,730.0 to 11,460.0.
FS 560.0 to 1,120.0.
[INAUDIBLE] to inspect the top region.
ANDREAS STEFIK: Now in quad trees, the brilliance of them
is that--
I mean, at least, the thing that I think is cool about them,
is that even when you get to a region,
you might know this has 11 points.
You might want to drill down a little further, so you can.
NVDA: Top left subregion-- top right sub-- bottom right
subregion 6 points.
VP 8,595.0 to 11,460.0.
ANDREAS STEFIK: So I'll stop it.
But the idea is that you can then
drill down to individual parts, eventually
to the points themselves.
NVDA: VP 9,025.0.
FS 680.0.
[INAUDIBLE]
ANDREAS STEFIK: And it gives you information
like what is this individual point, what
is the ordinality of this point in this particular region
and the cardinality?
So, like, one of six, how many points are in this region?
And you can traverse through them.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: And I won't go through every little detail
of the queues, but that's the idea.
So overall, the idea with making something like data science
more accessible to people with disabilities
is you need a mapping to the operating system
and you need to somehow provide structure for all of the items
that are in there, in addition to the other things we
mentioned with text and making the language easier to use
and stuff like that.
So I hope that--
I don't know.
NVDA: [INAUDIBLE]
ANDREAS STEFIK: I hope that this was interesting to you
and that you had some information that
might be useful to you when trying
to think about data science or to do it accessibly.
And at this point, I will open up for questions.
So thanks for coming.
BRIANNA BLASER: Folks, you can ask questions in the chat
or unmute yourself and ask them verbally.
STUDENT 1: Stefik, this is Richard.
ANDREAS STEFIK: Yeah?
STUDENT 1: Yeah, that was great.
I turned my video off.
I have a question about the distinction
between a Mac and the iOS.
Are there different accessibility approaches
to those two platforms?
ANDREAS STEFIK: Yeah.
So they use different architectures.
On windows, it's UIA.
On Mac, it's called NSAccessibility.
And on the one hand, they have a lot
of philosophical similarities.
You still do events and stuff--
STUDENT 1: I asked about iOS, not Mac.
ANDREAS STEFIK: Oh, on iOS.
I think they also use NSAccessibility.
But because of the touch system, I
think the events are different, if I recall correctly
off the top my head, so.
But I think if you add support for one,
you might not get the other for free, but you get a lot of it,
if I recall correctly, so.
STUDENT 2: Yeah, that's right.
They're very similar.
ANDREAS STEFIK: They're very similar.
Yeah.
I see a couple questions.
How difficult would it be to make
web graphs that are accessible?
Yeah, that's a good question.
So it's possible, I think, using stuff
like Shadow DOM and things like that
to make that more available.
But it is tricky.
It's one of the things we've wanted to do on our projects
for a while, is to add some support for stuff like that
into our accessible graphing thing.
But it's a little complicated.
You still have to add structure and you
have to make it traversable.
So that's really the key.
How do you end up doing that so that the screen readers can
hook in?
I've experimented some with Shadow DOM-style
implementations for this internally on the Quorum
project because we have a JavaScript version,
but we don't have anything deployed for that now.
So it's possible, but the web is tricky
because the accessibility is not quite as mature
as desktop operating systems are.
I don't know.
It works really well for navigating headers and stuff,
but for complicated graphics, it is a little funky, I guess.
I see Andrew shaking his head, so I'm
guessing I'm not the only one that thinks that.
But it's a little funny, so.
STUDENT 1: Stefik, some people have their hand up in the--
BRIANNA BLASER: Yeah, I see JooYoung has his hand up.
ANDREAS STEFIK: Oh, they do.
Yeah.
Please.
JOOYOUNG: Before everything else,
thank you very much for your awesome presentation, Doctor
Stefik.
That's really cool.
ANDREAS STEFIK: No problem.
And thanks for coming.
JOOYOUNG: Yeah.
I was wondering-- before everything else,
I, as a actual user of SAS graphic accelerator,
I think there is a way to navigate
scatter plot by cell by cell, meaning
that as quadratic layer.
So not only you can navigate from left to right or top
to bottom, et cetera, but you can navigate cell by cell.
So I think that's a very similar concept that what you
mean by structure navigation?
ANDREAS STEFIK: Mm-hmm.
JOOYOUNG: So I was wondering if you
are considering sonification on top of that structure.
Because right now, I guess, it would be really great
if we have spatial information added using sonification,
as well, at least a toggle switch.
What do you think about that idea?
ANDREAS STEFIK: I think it's a great idea.
I mean, there's quite a bit of research
on the use of sonification, like out of the International
Conference on Auditory Display.
And there's some other stuff, as well.
And on the graphics accelerator, as well,
I think they've done a really good job of trying to give you
the Gestalt feel of a graph.
And, I mean, their team might disagree with that.
I don't know.
But that's my impression, is that sort of stuff
is what they've kind of tried to give you.
I think it's good.
I'm not an expert on the literature,
but I do suspect that if we added stuff
like that in for data science, it would be useful.
I don't know exactly what context and when it would be
and when it wouldn't.
I'll give you an example.
One of the teams I work with does robotic telescopes
for young, blind people.
And so they have these robotic telescope arrays
and they get back these astronomical images.
And they've been experimenting with sonification there.
So for stuff like that, because it's not a scatter plot
where it might just go up, it's a lot harder
to figure out what the sonification tells you, right?
Are you sonifying globular clusters or spiral galaxies?
Does it sound different for each?
So I don't think it's an all or one.
I imagine sonification would have
to be used on kind of a case-by-case basis,
depending upon what you're trying to sonify.
But that's just my guess.
I don't really have strong evidence either way,
I just think it's cool and would probably work for some things.
JOOYOUNG: Can I kind of have piggyback question?
ANDREAS STEFIK: Sure.
JOOYOUNG: Yeah.
So I think auto-embedding accessibility
API, like you did, is really cool and necessary.
I think someone asked about web graphic, as well.
In my understanding, Jonathan Godfrey and their team,
they are aiming at SVG as a vehicle
to deliver like MathML type, zoom in and zoom out
navigation.
Actually, I have the experience.
There are some prototypes for histogram and box plot.
I think that would be great if someone
can develop some SVG plus auto zoom in,
zoom out, just like MathML.
So would there be any possibility for your team
to develop that support?
ANDREAS STEFIK: To the best of my knowledge,
the way that Jonathan Godfrey's stuff work is exactly
like you're saying it is.
It's basically like an SVG with some extra, basically,
structure information that gets tossed down.
I can't remember if it's embedded in the SVG
or if it's a separate file.
I forget off the top of my head.
But it's something like that.
The way that we've designed it is we've generated--
behind the scenes, we generate information
that basically matches to the kind of stuff
that Jonathan puts out.
So hypothetically, you could write a program
that could get that information and then send it off
as an SVG or some other format.
So there's nothing preventing it,
but we haven't done that ourselves.
In part, because we just invented this.
So we could do the operating system of hooks first.
And then however people want to use
it will be kind of based on whatever the community wants,
I guess.
So that stuff is absolutely possible, 100%.
JOOYOUNG: Awesome.
Thank you very much.
ANDREAS STEFIK: There's two other questions
I can answer quickly.
What other charts have you considered?
So far, we've done histograms and bar charts.
We don't use quad trees for either of those,
but my hunch is that quad trees will
be useful to a bunch of things, but we
don't know exactly which ones.
Gary, no problem.
Thanks for coming.
And Lev asks, has anyone used vibration features on phones
to help render graphs in a more accessible way?
Stronger vibrations as you drag your finger
over a larger cluster of points, no vibrations in empty sector?
I think some of the best work that
has been done related to vibration
is done by someone on the call, which is actually
Jenna Gorowitz and also Nick Giudice out of Maine
is a really fantastic scholar on this.
We actually just submitted a NSF proposal
on exactly what you just said.
What was it, a week or two ago, I think, Jenna?
Something like that.
And so we're going to try that.
I suspect that there is some serious potential there,
but it's kind of an open question in the literature,
I think.
Vibration is absolutely useful, it just
depends on for what and under what context.
Yeah, no problem.
BRIANNA BLASER: Awesome.
Well, thank you, Stefik, for the presentation,
and thank you everybody for joining us.
I'll be following up over email and asking
if you want to continue to receive more information
about webinars like this.
Thanks.
ANDREAS STEFIK: Thanks, everybody.