BRIANNA BLASER: I think we'll go ahead and get started
a couple of minutes after the--
I was going to say after the hour-- after the half hour.
So for those of you whom I haven't met,
my name is Brianna--
oh, I stumbled over my own name there--
Brianna BLASER at the University of Washington
with Access CS for All.
And so you may recall that we like
to get together every once in a while
to talk about accessible technologies for computing.
And so on Andy Stefik and Richard
Ladner suggested that they knew stuff
was going on with Blockly.
And so we wanted to reach out to those folks.
And so Abbie from Google is going to share with us.
Richard or Stefik, does either of you
want to say a few words about Blockly?
ANDREAS STEFIK: Sure.
You had asked me before the call to say something
I imagine many of the people on the call
know the Blockly stuff already.
But Blockly became sort of the de facto standard
for converting a programming language to blocks online
and a lot of groups adopted it.
I think co.org did.
I know that there's some Python stuff out there and others
as well that kind of run on top of Blockly.
So I remember at I think it was VLHCC last year.
There was some conversations with people on the Blockly team
about them doing some accessibility work
because if that becomes accessible,
then a lot of the downstream tools that use Blockly also
kind of become accessible as well, which is a good thing.
So that's one reason sort of why we thought
it might be cool to follow up and see what's going on,
what they're working on, and stuff like that.
I mean, that's the scoop.
BRIANNA BLASER: Well, excellent.
Well, without further ado, Abby Schmidt from Google is here,
and I'll let her take it from there.
ABBY SCHMIDT: Hey, thanks.
OK, I'm going to share my screen or attempt to share my screen.
And my Zoom skills are not great, but--
all right, cool.
So yeah, I have a presentation.
Feel free to interrupt at any point and ask questions.
If something doesn't make sense, I can try and answer
any questions you have.
OK, cool.
Can everyone see my screen?
ANDREAS STEFIK: Yeah.
BRIANNA BLASER: It looks great.
ABBY SCHMIDT: Awesome.
OK, so yeah, my name's Abby.
Work on the kids coding here at Google.
We primarily maintain the Blockly library.
And today, I'm going to talk about how
we offer keyboard navigation support to our Blockly.
So yeah, I think this is what you already basically covered,
but Blockly is a JavaScript library for building
visual programming editors.
So what this looks like is Blockly
gets used in all the applications
or in a lot of applications that teach kids
how to code using blocks.
So this is great but not exactly accessible
since it's so visual.
So today, I'm going to give some background on what we've
tried in the past and then talk about where we are now
and give them a demo of what we currently have.
And then I'm just going to talk a lot about all
the different ways that you can modify keyboard navigation
and then finally some future work.
So some background.-- So in the past,
this is actually me before my time at Google,
but they had tried to take the Blockly workspace, which
is what users used to write all their code
and connect their blocks together,
and the tool box, which is where users get
the blocks to write their code.
And they tried to create a purely,
like, table that had all the different blocks in it,
and the blocks are represented as the buttons and text.
And so they came up with a demo for this,
and they tried it out.
And one of the problems they found
is that it was actually really difficult to navigate
using a screen reader.
So since our target audience is primarily students,
this was something that they really struggled with,
and so we didn't keep continuing down this path.
So then a few years later, we ended up
on keyboard navigation.
So the reason we chose keyboard navigation was it
was kind of our first step in creating a Blockly
experience that was more accessible for people
with visual impairment.
It was also useful for sighted users as well,
and it was potentially useful for people
with motor impairments.
So this is kind of like our first step.
And then our goal for keyboard navigation
was to create an easy way for developers and researchers
to test keyboard navigation.
So we understand for keyboard navigation
that we don't necessarily have all the answers right now,
and there's a lot of things that aren't completely
straightforward and that we don't know.
So we really wanted to create a good way that
allowed other people to test what would be the best
practices in what would become the best
practices in this space.
So now I'm going to give a quick demo
of what keyboard navigation currently
looks like for Blockly.
If at any point I'm going too fast, let me know.
It's kind of hard to show everything
I'm doing with my screen or with my keys.
So this is a Blockly workspace.
If I hit a Control-Shift-K, this is
going to enter us into keyboard navigation mode.
So as you can see, there's a cursor
on the screen that's blinking red.
When I hit Enter, this is going to mark this location using--
so it shows it marked with a blue marker in its spot.
And so now I can just enter any blocks into this space.
So if I hit T, that's going to open the tool box.
And this is where all the users get their block
that they're going to use to write their applications.
And I'm going to use the WASD keys to move around the tool
box and those blocks.
RICHARD LADNER: Abbey, this is Richard.
I just want to interrupt you for a second.
Just want to remind you that not everybody in this meeting
can see what you're doing.
So--
ABBY SCHMIDT: Right.
I'm trying to explain it as best I can.
But if anyone has questions, please let me
know, and I will do my best to explain it better.
Yeah, so by hitting the D key, this takes me to a block
that I can put onto the workspace.
So in order to add a block to the workspace,
I can get the Enter key.
And this moves the block from the tool box
into the workspace.
So now I can navigate to different-- each block has
different connections on it.
And so I'm going to want to add a block after this block.
So in order to do that, I can navigate to the next connection
point on this block.
So I'm going to do this using the S key.
So when I hit the S key, it goes to the next connection
point on this block.
And when I hit the Enter key, it's
going to mark that location.
And that will allow me to insert something after the block
that I have placed on the workspace.
So I'm going to hi the Enter key,
and that marks that location.
And I'm going to go back to the tool box which
holds all of the blocks by hitting the T key
and navigate around to find the block I want.
And I can do that using the S and the D key.
So now I've entered the block.
I'm skimming over a little bit how
I am navigating to the different connection points and stuff
just because I'm going to go over that a bit more later on.
CLAYTON: Thank you, Abby.
Question?
ABBY SCHMIDT: Yeah.
CLAYTON: Is there now or are you imagining
that there might be accompanying audio feedback along
with these actions and effects?
ABBY SCHMIDT: Yeah, definitely.
So that's definitely for the future work section of this.
Right now, we just have keyboard navigation support
and we have looked at a few different ways
to offer screen reader support as well.
But we just haven't gotten there yet.
CLAYTON: Yeah, I wasn't necessarily thinking
about screen reader but something that would tell you,
for example, you have selected the after connection or the
before connection or whatever so that you
get something in addition to the visual indication of what
you've done.
ABBY SCHMIDT: Got you.
Yeah, it's definitely something we'd like to look at.
We just haven't gotten around to it yet.
CLAYTON: OK.
ABBY SCHMIDT: Yeah.
So-- and that's also another area
that we're expecting people to experiment in.
Just because we're not entirely sure
what wording would be best for things
like the next connection.
Like, does the student understand what that means?
CLAYTON: Mm-hm--
ABBY SCHMIDT: So yeah.
OK, so to remove a block or to disconnect blocks,
I can hit the X key, and that will just
connect those two blocks.
And that's basically what we have for keyboard navigation.
And so there's a few things here that can be--
I'll elaborate more on.
But does anyone have any questions on that?
No?
OK.
So the rest of this presentation is just
going to talk about modifications or different ways
that we expect people to experiment
with keyboard navigation.
So like I said earlier on, we're a JavaScript library.
We're not the end application here.
So everything that we make has to be easily modifiable
and easily changed to fit all of the different use cases and all
the different users' needs.
So one thing we expect people to mess around with and change
a lot is key mappings.
So we wanted to provide an easy way for users and developers
to customize the keys that users use for each key navigation.
So what that really means is for this demo,
I was using the WASD keys to do things like previous, out,
next, and in.
However, we are aware that that might not work for everyone.
So by giving developers an easy way to change this,
it allows them to surface that easily to their users.
So that's an easy thing we expect people to want
to change and experiment with.
Another thing we expect people to change and experiment with
is adding shortcuts.
So for example, if you hit the C key,
you might want to collapse all your blocks.
If you hit Shift and S, you might
want to go to the next stack of blocks on your workspace.
We're not entirely sure what all those shortcuts could be.
There's a lot of them, but we're not
entirely sure which ones are going
to be the best for everyone or the best set
to offer to people.
And then finally, just change how the cursor looks.
So this is important because some
of the connections on the blocks are difficult to display
or difficult to show kind of what they mean.
So we tried a few different scenarios
of what would be the best way to show these different cursors
and what they mean.
And we settled on what we came up with for the demo,
but we didn't do a lot of experimentation around it.
So that's another area we expect people to experiment with.
Also, people have different shapes blocks,
So for every different renderer they have.
Then they will need to create a different way
that the cursor looks that will fit to those blocks
that they have created.
And kind of the heart of this matter is workspace navigation.
So this is kind of the most important part--
for me, at least-- is that how does
someone navigate all of the programs
that they have on the screen.
So as you can imagine, kids, when they're writing code,
can have lots of different huge applications that have,
like, hundreds of blocks in them.
So how do we make it so that a user who is visually impaired
or a user who has motor impairments
can easily read through those stacks of blocks
and navigate around all of the different pieces on those?
So right now I'm sharing a screen is
a few stacks of blocks on the workspace,
and they have red lines around them.
This is just showing that a workspace is made up
of different stacks of blocks and also
the different coordinates on that workspace.
So all of these are something that a user
is going to need to be able to interact with.
And then in each of those blocks,
there is a lot of things that a user can interact with.
So this is kind of confusing, and I understand that.
But it's basically a block, and it
shows all of the different locations
that the user can connect with.
And not only all the different connections
that they can connect with, but there's also fields on blocks.
And so fields are things that a user can change.
So for the block on the screen right now,
it says repeat 10 times.
A user can change that tend to be repeat 8 times, 7
times, whatever they want.
And so on each block, there is a decent amount
of things that need to be interacted with in order
to make Blockly functionable and usable.
We created-- in order to organize
all of these components, we created an AST.
So the Blockly AST holds every field, input, connection,
block, and stack and workspace for Blockly.
So this gives developers an easy way
to at least understand how all of these things that we've
created or that are on the workspace that need to be
interacted with are organized.
So I have this picture, which is very confusing
and no one really likes.
But it's a tree.
And it's not really important to understand
exactly how the tree looks.
There's more documentation for that.
If anyone is interested, I can share that.
But what is just important to understand
is that a tree exists that holds all
of these elements that need to be able to be interacted with.
And so we created the AST as the bottom layer,
but on top of that is the cursor.
So the cursor is what the user is actually
going to interact with.
And it controls how the user navigates around that AST.
So we understand that not everything in that AST
might be something that needs to be navigated to by a user.
It needs to be navigated to by user,
but maybe there's different situations
where you want to change how someone
navigates around that tree.
So the cursor sits on top of it, and you
can create cursors that navigate the tree in different ways.
So does anyone have any questions so far?
No, OK, cool.
So I'm going to just demo real quick some other cursors
we have.
Let's see-- some other cursors that we have created.
So the default cursor that we have
navigates very similar to how the AST is set up.
So this might be great if you have someone
who is very advanced and can understand
how that tree is set up.
So when I hit the S key, it goes down
to the next connection or the next block.
And if I want to go inside of the block, I can hit the D key.
So this is great.
It might be faster if you actually understand the tree,
but you can imagine if you have a younger audience,
then this is not going to be great
because they're not going to understand
how that tree is set up.
So we've also created something called a basic cursor
because naming is hard, and this is the most basic cursor I
could come up with.
But this one navigates everything
that you've seen in that tree by doing a preorder
traversal on that tree.
So I can get to every element that I
need to just by using the A and D keys.
So this is good for some, maybe if you
have a new user that doesn't understand how the tree works
or just wants to get to all of the elements in the tree.
But you can also imagine that this is very slow because you
have to navigate one by one through everything
that you see.
So that is the basic cursor.
That's one way we can see a cursor being implemented.
Another cursor we've tried is what I'm calling a line cursor,
and it just goes through the blocks
as if they were a text editor, basically.
So by hitting the S key, I can go to the next line,
basically, in the code.
And then by hitting the A and D keys,
I can go to, inside that line, everything that is--
all the fields and inputs and connections that
are inside that block.
So these are just examples of cursors that people can make.
You can also make a cursor that only goes to blocks or cursor
that only goes to connections.
You can imagine if you wanted to do something like block mode
where you go to a block, and then you
can insert in edit mode or something like that.
Yeah, so that is the cursors.
RICHARD LADNER: This is Richard again.
Abby?
ABBY SCHMIDT: Yeah
RICHARD LADNER: Very impressive.
Have you thought about switch users, not just
screen reader users?
Because the way you navigate there,
either-- you know, the hierarchical way
of doing it or the other ways.
You know, these are different ways
that you can navigate using a switch.
And so that might be-- and switch users
are used to having hierarchies because that's quicker.
So if you haven't thought about that,
maybe you should put that in your work.
ABBY SCHMIDT: Yeah, for sure.
We haven't-- like I said, it's a little bit difficult for us
to do testing and stuff like that because we are the library
and we don't have a lot of end users.
But that is definitely something we've looked at or considered
is the fact that it would be great for people with motor
impairments or switch users.
So yeah.
OK, and feature work, which is that there's still a few things
on the workspace that cannot be gotten to you by that AST that
we have created.
So there's things like a trash can that users can throw away
their blocks in, workspace comments on icons on a block,
and a few other things.
So those we'd like to support in the future
to make sure that the AST holds everything that a user needs
to be able to get to.
And then obviously the most important thing
or one of the most worrying things
is how to communicate the cursor's position.
And this goes directly with also screening or integration.
So we've looked at a few ways to add screen reader support.
One of them
Stephanie Ludi was using was updating aria-live region
whenever the cursor moves.
And so it would read out the text
or wherever the cursor currently is to the user.
The other way that we had looked at
was updating the label on the cursor
since the cursor is an SVG element.
We could update the cursor and then set the focus
on the cursor as it moves.
So I have a little GIF, and it's just
showing every time I move the cursor,
focus also moves along with that cursor.
Yeah, so there's resources which are links
but I did not share on my side.
So I can share those out later if people would like.
But basically, there is documentation now and also
a code lab for how to work with all the APIs
that we have created.
And then there's also a demo page
where users can play around with keyboard navigation.
So any other questions?
PAMELA: Hi, this is Pamela from Khan Academy.
ABBY SCHMIDT: Hi.
PAMELA: Hi.
I'm just curious.
Could you demo how you would--
how a user would basically reorder to blocks?
ABBY SCHMIDT: Yeah, for sure.
OK, when you say recorder, do you mean they would--
PAMELA: Like, I'm trying to see you
like how dragging is translating in the keyboard world.
So if you had--
like on the keyboard navigation demo,
you've got multiple blocks inside the if, I think.
So maybe if you wanted to move one of them over the other.
ABBY SCHMIDT: Sure.
So right now, we have--
the way we're doing this right now is like, we have a marker.
So I can move the main cursor around,
and then I can mark a location.
And then I can go to the connection
and I want to move to that location.
And for this, I can hit I. So that will move that there.
If I want to add something inside of this if block,
I can navigate to that internal-- this statement
input.
So I can mark this location here and then
I can navigate to the connection.
I want to move there.
So then I navigate to this repeat block, and if I hit I,
it will insert it inside of that if block.
Is that what you're asking?
Or--
PAMELA: Yeah, yeah, that gives me a lot more insight.
ABBY SCHMIDT: OK.
Are there any other questions?
Seems like Clayton's talking, but--
CLAYTON: Yeah, that's [INAUDIBLE] sorry about that.
ABBY SCHMIDT: No worries.
CLAYTON: Yeah, great work.
It's terrific that you're doing this.
Are you by any chance in touch with Sina Bahram of--
do you know him or do you connect with him at all?
ABBY SCHMIDT: I have not.
CLAYTON: Yeah, I can try to make an intro,
and I'm sure others can as well.
The reason he comes up it is that one of the things that he
talks about is--
and this relates to one of the questions I had earlier.
So when you're thinking about audio information,
there's possibilities beyond screen readers
that one might want to think about.
So it's sometimes called making things self-voicing.
And this can give you freedom.
So screen reader navigation tends
to be very hierarchical as you're aware,
and you've been playing with various things already.
That's not always the best way to do things.
So sort of spatial ways of navigating,
sort of arrow key types of things, which really
are sort of native to the screen reader world,
can make sense for the kind of thing that you're doing.
So I think, you know, Sina might have some interesting thoughts
about, again, possibilities in addition to screen reader.
I'm not saying you don't want to have screen readers support.
But for some users, giving them good audio information
without having to use a screenwriter
might have some advantages.
And--
ABBY SCHMIDT: Yeah.
CLAYTON: I've got to break off, but thanks
for the presentations.
BRIANNA BLASER: And-- this is Brianna.
I can make that introduction to Sina afterwards, Abby.
ABBY SCHMIDT: Yeah, that sounds great.
SPEAKER 1: And that self-voicing option,
it was actually listed in one of her slides.
That's the aria-live region approach.
And that does work, but that's what an aria-live region is.
It dispatches speech through what's
called a UAA notify on the operating system level.
What that means is that it sends text to the screen reader
directly and asks it to request speech.
But that's important because if you just do pure self voicing,
you'll actually get two things speaking at once when
you're using a web browser.
So like, you'll be using Skype as a blind person.
And then you would hear Skype blabbing at you,
and then you'd also hear this other speech
in the Blockly interface.
So aria-live regions is the appropriate way
to do that if you go that approach.
CLAYTON: Yeah.
ABBY SCHMIDT: OK.
CLAYTON: OK, thanks, all.
RICHARD LADNER: Bye, Clayton.
ANDREAS STEFIK: Bye, Clayton.
Sina Bahram also published a paper
on doing these kind of allegations with Schanzer
out of Bootstrap last year.
And there was one other paper that
was published on these shortcuts as well that you
could consider looking at.
I'm going to post that the editor.
RICHARD LADNER: OK.
ANDREAS STEFIK: That's the frame-based editing
that Neil Brown and his team out there, Michael Colling did.
It's not so much that their stuff
is screen reader accessible.
It's just a different way to handle the shortcuts.
And they do actually have some evidence on theirs
which I thought was interesting, as does Emmanuel Schanzer.
Personally, I think Schanzer's approach is probably
a little bit easier to use.
That's just my personal opinion.
But both approaches are plausible.
Like, if I had to copy one, I'd probably
copy the one from Bootstrap off the top of my head.
But I'm just making you aware if you don't know those papers.
So--
ABBY SCHMIDT: Yeah, I don't.
This is great, all great information to have.
ANDREAS STEFIK: I'll see if I can't get the other paper
posted in there to see--
BRIANNA BLASER: I just dropped it
in Stefik this, Bootstrap SIGCSE one.
Yeah.
ANDREAS STEFIK: Yeah, I think that--
BRIANNA BLASER: OK.
ABBY SCHMIDT: This is great.
Thank you so much.
ANDREAS STEFIK: Yeah, that's the one.
Yeah.
There's also the Blocks4All stuff.
I don't know if you did stuff on the keyboard.
I think you did, Richard, but it's been a while
RICHARD LADNER: No.
ANDREAS STEFIK: It's touch screen, right, for yours?
RICHARD LADNER: It's touch screen.
Yeah.
ANDREAS STEFIK: Yeah, so that's different.
RICHARD LADNER: Blocks4All, it's just touchscreen only.
ANDREAS STEFIK: Yeah.
RICHARD LADNER: It only uses a screen reader.
It doesn't use self voicing.
So it just uses the Apple infrastructure to do--
for development.
ANDREAS STEFIK: Yeah, so when do you do add screen reader
support, the one thing that I would also recommend
is if you have to choose between the two choices,
between an aria-live region and doing the focus,
I would recommend doing the focus.
And there's a really good reason for that.
And that is because as the developers,
you can control the focus with some good defaults.
So what that means is that as the system goes,
you can control, like, what happens
when you press up and down.
And then there's systems that you
can follow that have been shown to be understandable.
Specifically, there's two papers, one by Katie Baker,
which was one of Richard's students back in the day.
And that was on--
I forget the name of the paper.
But basically, it showed which parts of the AST
are easy to navigate to.
That's one thing.
So it is actually quite comprehensible for people.
And number two, the study was actually
replicated by Ameer Armaly, who's a blind guy that
was out of Notre Dame.
And if you adjust using standard focus methods,
it actually is pretty easy to use for most people.
So I wouldn't assume that you have to do something fancy.
I would actually make the standard focus and navigation
thing to be the default. And then
if you want to allow people to adjust it
through an aria-live region to experiment, that's great.
But having some sensible defaults
that take into account a couple of those papers
would be probably 90% of the problem is what I would just.
RICHARD LADNER: Do you mean the Struct Jumper?
ANDREAS STEFIK: Struct jumper, yeah,
because then, it let you look really
carefully which things you should
be able to jump to, right?
And that was, I think, what was so compelling in her stuff.
It's not blocks.
It's different.
But nonetheless, the idea is very similar.
So--
ABBY SCHMIDT: OK, that sounds great.
BRIANNA BLASER: Got to that one in the chat as well.
ABBY SCHMIDT: OK.
Perfect.
ANDREAS STEFIK: This is great, though.
I actually really appreciate you guys are doing this work
because it's--
I can see how much progress you've made over the last lord
knows how long.
RICHARD LADNER: Yeah, Abby, did I meet you
at Blocks and Beyond?
I don't know.
I can't remember.
Were you there?
ABBY SCHMIDT: Yes, mm-hm.
RICHARD LADNER: I met you there.
Oh, great.
ANDREAS STEFIK: I think we had lunch too.
I'm trying to remember exactly.
RICHARD LADNER: Well, you weren't there at Blocks
and Beyond, [INAUDIBLE], so--
ANDREAS STEFIK: No, no.
I could've sworn that--
I can't remember if it was Abby or someone else on the team.
ABBY SCHMIDT: Yeah, VLHCC.
ANDREAS STEFIK: Yeah, VLHCC.
Yeah.
I think so
ABBY SCHMIDT: Yeah, I think we did have lunch.
ANDREAS STEFIK: Yeah.
RICHARD LADNER: A question, sort of a high level question, Abby.
ABBY SCHMIDT: Yeah.
RICHARD LADNER: So these areas, they're
on the screen in different parts.
And you know, how do you make sure they don't overlap,
or how do you--
if you're blind, you can't tell how big they are exactly
and how much space they take.
So how is that going to work?
ABBY SCHMIDT: So are you talking about the blocks on the screen?
RICHARD LADNER: Yeah, the blocks-- not just the blocks,
but the segments of code, have multiple segments
of code, or even the blocks.
ABBY SCHMIDT: So you're saying, like,
if you have different, like, stacks.
RICHARD LADNER: Yeah, but not just one, but like many,
several.
ABBY SCHMIDT: Yeah, I'm trying to show,
but I'm not doing a job.
RICHARD LADNER: Oh yeah?
Really cool demo, by the way.
ABBY SCHMIDT: Yeah, thank you.
so you're saying if you have multiple stacks--
RICHARD LADNER: And those blocks are separate
and you're blind, how do you place these things?
ABBY SCHMIDT: Yeah, it's a good question.
So we hadn't thought about that that much.
If someone is visually impaired, then
if they're sharing the code, I think
this is where the problem would arise.
If they're sharing the code with someone who is sighted
and who needs to see the different levels,
then that becomes a problem.
If they're just using it for themselves in order to read,
it's less of a problem.
But yeah, that's a good question-- something
to think about.
There is a way we could possibly split up the workspace
into different regions.
And so maybe if they hit a key, it
takes them to a different region where
they can write another stack of code or something like that.
RICHARD LADNER: Yeah, I notice for putting those things out
there in the first place, you dragged and dropped,
which is probably not very accessible.
Is there a way to just put something out there
and it'll just go to the right place?
ABBY SCHMIDT: Yeah, here.
I can do that right now.
So--
RICHARD LADNER: Without naming the place, though.
You know?
ABBY SCHMIDT: Without-- what?
RICHARD LADNER: Well, right now, you
have that blinking cursor, which is where
you want to put something.
ABBY SCHMIDT: Mm-hm.
RICHARD LADNER: But suppose you don't have that.
You could just take a block and put it on the workspace.
ABBY SCHMIDT: So the cursor is how
we're currently making it so that people
can add to the workspace.
RICHARD LADNER: I see.
ABBY SCHMIDT: Yeah, so the cursor, it
could, if someone is visually impaired,
have it so that it speaks out where it currently
is on the workspace and maybe say some information
whether it's on top of a block or not.
But the cursor is what we would use.
You can hit enter, and then that would mark the location.
And then you can enter or add a block from the tool box
on there.
ANDREAS STEFIK: The cursor approach
is exactly the same as the Stride team as well.
That's why I mentioned it-- because they do--
ABBY SCHMIDT: OK.
ANDREAS STEFIK: --the same way.
So.
RICHARD LADNER: Oh, in Blocks For All, there is no cursor.
ANDREAS STEFIK: Yeah.
RICHARD LADNER: So you just-- there's only one block.
And you can edit that block sort of in a similar way
you described earlier.
And so there isn't this issue of this sort of space
that you have to know about.
It's all in one place.
It's all at the bottom of--
ABBY SCHMIDT: Yeah, that is definitely a talent
that Blockly has.
Yes, especially since we are something that, like I said,
the library and not the end user so we
can't force people to put blocks in a certain location.
So anything that we have currently,
we have to kind of adapt in order
to make it more accessible.
ANDREAS STEFIK: If I remember correctly,
the data from the Stride team-- again, this isn't my data.
This is just somebody else's data.
ABBY SCHMIDT: OK.
ANDREAS STEFIK: But if I remember correctly,
they did some, like, post-interviews with people
on like the kind of problem they face
when doing the keyboard part.
ABBY SCHMIDT: Mm-hm.
ANDREAS STEFIK: And the user feedback was basically that
like, because it's not the typical what you would do
in a text editor-- like, type, type, type, type, type,
and it has standard keys--
A really means literally A--
it seemed like people had difficulty figuring out--
sort of learning what the keys were, right?
It felt like there was a learning effect that went on.
That was my impression from the data.
That didn't actually stop their data
from showing evidence that it was slightly
easier to use than text.
You made less compiler errors and stuff.
So that's still OK.
But nonetheless, it might be that maybe the thing
to look at when you do testing is
to try to measure that initial learning curve over--
as opposed to users that are already familiar.
In other words, focus on lowering the learning curve
as opposed to the later curve would be I guess.
ABBY SCHMIDT: Yeah, that's probably very good advice
success, especially since a lot of the people using this
are going to have a high learning
curve if they're younger.
ANDREAS STEFIK: What I was thinking.
The other thing too is that because of that data
from the Stride team, the one thing you might consider
is seeing if you can't map the navigation keys that we use
to the ones that you would just press accidentally if you were
like the normal blind user.
For example, in a normal text editor,
you wouldn't use WSAD to move around.
You would use Up, Down, Left and Right, right?
I don't know if that would actually
be more intuitive because I've never run a study.
But I would guess that it's the type of thing someone
would do initially.
And as such, you might lower the learning curve
by using the more standard text editor keys
to move around instead.
It's just speculation from data.
But like, you know, that's my speculation.
So--
ABBY SCHMIDT: Yeah, that's definitely fair.
I've talked to someone who was a new discriminator,
is new to using keyboard navigation who
is visually impaired.
And she did-- that was her one thing.
She's like, I've used a few other text editors.
And it would be nice if the same keys
were used between these two.
ANDREAS STEFIK: Yeah, that's exactly what I figured as well.
And people said that in the study that Neil Brown had too,
that they couldn't figure out what the magic keys that they
chose were.
So WSAD is interesting.
He must have mapped it off of video games
is what I'm suspecting.
SPEAKER 2: [INAUDIBLE]
ABBY SCHMIDT: Yeah, so we chose WASD
because I think we were told that using the arrow keys
doesn't work with screen editors.
SPEAKER 2: Last thing we heard was
that you were prompting or asking
who ran a global cloud set.
ANDREAS STEFIK: I don't know if that was intended for us
or not.
ABBY SCHMIDT: OK.
ANDREAS STEFIK: I'm sorry, what did you say?
It was the-- oh, Up, Down, Left or Right
doesn't work with screen readers?
ABBY SCHMIDT: Yeah, like, if we're not using focus,
I guess, or if we're not changing focus,
then using the arrow keys is already used by screen readers
to move around the screen.
So I also tried-- go ahead?
ANDREAS STEFIK: That might be true,
but there's lots of ways around that stuff.
So that's not usually a problem.
Think about it like this.
Like, the screen reader pipeline,
when it comes to like navigating DOM,
basically, as you adjust the focus,
it usually sends off an event.
You can track that through these specialized tools
that I can show you another time.
But in addition to that, you can always
issue one of these aria-live requests.
And that forces the screen reader to talk.
So it's like, think about it like this.
Like, as if you do some keys and they
don't do exactly what you want, you
can still say, hey, screen reader.
I'm going to force you to speak.
And that's going to issue a command
exactly how I want regardless of what
you think you should do here.
And so you want to use those sparingly.
Like, you kind of want to know what you're doing.
But like, there's absolutely fullbacks that do work,
and they work correctly.
So--
ABBY SCHMIDT: OK.
ANDREAS STEFIK: Ish.
Technically, the aria spec is slightly broken,
but never mind that.
So--
BRIANNA BLASER: I see that JooYoung has his hand raised.
So I want to give him a chance to ask a question.
AUDIENCE 1: Hello.
This is JooYoung.
Can you hear me?
ABBY SCHMIDT: Yep, hi.
I can.
AUDIENCE 1: Yep.
Before everything else, thank you so much
for such a nice presentation.
I'm blind myself.
I'm a blind programmer.
And I also did some research on block programming
using tangible physical blocks called KIBO.
And the reason why we say drag and drop block face programming
is not accessible, I think we have two reasons--
first, because screen reader users,
we have two different modes-- input and output.
In JAWS or MVDI, we have virtual cursor and forms mode.
So when we navigate the DOM trees,
we virtualize the DOM trees so that we
can navigate the hierarchy.
And when we type something in text area,
we use standard keyboard navigation.
So these different interactions between input and output,
focus mode versus document mode, or virtual cursor
versus forms mode, it splits our screen reader behaviors.
So sighted people can see and type simultaneously,
but in order to navigate the code,
we have to go back to virtual cursor.
And it sometimes prevents blind people
from understanding what's going on with the screen
simultaneously and quickly.
So aria-live region is really good way,
but that's spoken one time.
So in order to navigate the output character by character
or word by word, in turn, we have
to turn back to virtual cursor.
So I was wondering if you can implement document role or aria
role equals document for output reason
so that screen reader can be forced
to use the virtual cursor automatically,
and aria role equal application for input area
so that we can use focus mode.
So for novice screen reader users,
they can't choose between those two different modes
because they are--
because these kind of different modes
require some kind of experience.
And it's not very easy to understand for young people.
So that's one suggestion and concern.
And second, very-- I'm sorry.
Firing off too many questions an ideas.
ABBY SCHMIDT: No, that's great.
AUDIENCE 1: And I think blind people cannot fully benefit
from drag and drop and block based programming because just
only have to listen one line at a time,
have a cognate load in the short term memory.
But sighted people, they can visualize the construct
of 3D modules on the screen.
So I was wondering if you have any plans to have
physical blocks for Blockly slot objects
so that young people can tangibly
understand and experience embodied learning
before this digital access.
ABBY SCHMIDT: OK, great.
Thank you for the questions.
So the first question, I have not looked that much into it.
I think it said aria-live roles set to document.
And I think input was the second one.
AUDIENCE 1: Yeah, for output area,
we use role equals document.
That forces screen reader to turn on virtual cursor
or document cursor.
That's the required mode for screen reader
to parse the screen output.
And role equals application, that forced the screen reader
to turn off virtual cursor, which uses standard keyboard
navigation.
So you said you your team assigned ASDF?
ABBY SCHMIDT: W--
AUDIENCE 1: W?
AS-- yeah.
Anyhow, that conflicts with screen reader pre-assigned
keyword.
So that should force screen reader
to turn off virtual cursor, which can be achieved
by role equals application.
ABBY SCHMIDT: OK, great.
I haven't looked a lot into those,
so I will definitely do that, which is great.
And then also, we have the different key mappings,
and we try and make it easy for people
to set key mappings so that if it does interfere with how they
currently have something set up, they'll be
able to change it is the hope.
And then for your second question,
we've definitely seen some stuff around physical blocks,
and we think those are great solutions.
The problem is that a lot of places that use Blockly cannot
necessarily afford those.
And so having Blockly--
trying to make locally accessible
is trying to kind of address that need
is that there's some places that just can't afford
that technology that Blockly, since it is free
and is accessible to basically everyone who has a web
application, it just makes it more
accessible for a lot of different people
and different user groups.
So by trying to make it accessible,
we're trying to also make it accessible for all
of those other people as well.
AUDIENCE 1: I see.
Would you mind adding 3D sonification for the block
output area so that we can understand
the dimension of code formation and structure?
So a block on the left side or a block on the right side
can be listened in this 3D audio.
ABBY SCHMIDT: OK, that's not something,
again, I have looked that much too since we've just been
focusing on keyword navigation.
But it's definitely something I will look into more and--
yeah, we'll look into more in the future.
AUDIENCE 1: Thank you so much.
ABBY SCHMIDT: Yeah, of course.
Anyone else have any other questions or just suggestions?
Resources?
BRIANNA BLASER: OK, this is great.
I think we're all really glad to see this work being
done and excited for this might be what this might lead to.
Thanks, everybody, for joining us today, and especially thanks
to Abby for putting this together
and sharing what you all are doing.
ABBY SCHMIDT: Yeah, thank you, everyone, for the feedback
and everything.
I really appreciate it.
ANDREAS STEFIK: Yeah, thanks for coming.
We appreciate that too.
RICHARD LADNER: Yeah, thank you.
Thank you, Abby.
ABBY SCHMIDT: Thanks.
BRIANNA BLASER: Thanks.
AUDIENCE 1: Thank you.
ABBY SCHMIDT: Bye, folks.
Bye.