All right good afternoon everyone it's lovely to see you on such a beautiful day for all getting here. Yes it's our very wintry installment of the S.T.P. lecture series I think I know most of you but those of you I don't know I'm Joy Rody and I'm the interim director of the Science Technology and Public Policy Program here at the Ford School of Public Policy today of the great pleasure of introducing our speaker who we dragged here from San Francisco. As she said she's getting the full Michigan experience but both of them regarding Feb 15th. the next round of the admissions to the certificate program are due Feb 15th the link to information is on our website and you can find that on the program for today also on Feb 15th Tom collegial who is currently the chief innovation officer Schmidt futures and formerly an S. and T. policy and also an economic advisor in the Obama and Clinton administrations is going to be giving a lunchtime talk that's a Friday so he'll be speaking at lunch and US Science and Technology Policy past present and future so you might want to put that on your calendars but now for the main event I am delighted to introduce Kerry Bishop She is the chief digital services officer for the city and county of San Francisco and as you'll see from her bio she believes passionately about public services designed she believes passionately that public services should be designed around the people who use them she began her career in local government in the U.K. She's worked in Europe and Australia before coming to San Francisco in 2017 she spent 8 years running future guy which is a London based digital design agency for public services really been on the vanguard. Of of sort of public technology. In San Francisco she is leading the team that builds new digital services and redesigns services for folks in the city of San Francisco today's tack is co-sponsored by the school of information and also supported by our student group inspire So thank you all for your support and after the talk there will be a Q. and A You should have no cards Come around and pick those up starting around 415 write your questions on them and R S T P P students and inspire leaders Lindsay Machar and Laura Greer will be helping us with the Q. and A today assisted by her S.T.P. program manager Molly Kleinman So without further ado please join me in giving a warm welcome to Carrie Bishop thank you. Everyone thank you for braving the weather to be here I really appreciate you showing up in such force at apologize I'm getting ever cold so I'm going to be like drinking and kind of. Hopefully not coughing too much through this but bear with me. So carrion thank you for the really generous introduction. So yeah I guess I'm going to just talk a little bit about my experience is like how I got to where I am what I'm doing today and kind of some of my observations and thoughts about working in the kind of intersection of technology and public service and I I haven't really majored on emerging technology and like how we deal with things like. Sidewalk robots but but I know that that's what some of you have been interested in and so I'm really happy to talk about that stuff in the Q. and A because I think it's I've heard from a few people today that that's kind of area of interest so this isn't really come about but I think we can definitely get into it in the Kinect that's what you're interested in talking about so kind of. If you would I prepared so this was the 1st place I. Worked this is the 1st directing this is the London bar of planet So London is so big that there's like a 1000000 people there so they break London into multiple municipalities smaller municipalities and each one has a population of around 300000 so than it is in the northwest of London this is the 1st office that they had I think this might actually be older than America or they're not fully show of the date of the building but. So and I was a graduate student so I had finished my undergrad I did work for a year in a charity and then I applied to be a. Graduate on a program that is the U.K. is kind of program for local government management so they recruit people out of college and stick them on this program for you kind of try out different you have placements new try out different placements in the city every 6 months you do rotation alongside that you kind of study 1st post-grad so I did that for another 2 years while I was there did also different things but the main thing I majored on was change management and so what we were doing at that time and we're talking this way too but we're talking around $50000.00 for. What we're doing at that time was implementing big I.T. system change in the organization so we were kind of taking all these small databases and Excel spreadsheets and kind of bits of paper that the city was running on we're consolidating that into one huge I.T. system and it looks like crap so these are like screenshots of the port types of products that are using both as a peer which is this big financial system and I often call out S. and P. because that's the one that we are implementing but actually the other there's only really one other competitor in the market and that's by Oracle and that's equally as terrible so like just to honor how terrible they both are this is what they look like as you can tell they're the user interface. It is horrific it makes the eyes bleed any look at it and you need a 3 day training course to understand how this thing works and you know my role there was a kind of implement a change of make people feel super positive about using this new I.T. system like they mostly they wanted to kill me especially and this was kind of kind of. Really for me when I started to think about like what am I doing here and what is my kind of career trajectory especially because I was talking with social workers and people who are really out there on the front lines of public service but how they needed to change the way they were working in order to work with this technology system which as you can see looks like it was designed in the dog so it really felt like I wasn't doing any good for public services because I was making us change the way we were delivering our services for technology and this was of course in 2004 about the same time as Twitter was becoming a thing and Facebook was becoming a thing and you don't need a training course to understand how that technology works so why is it that public service the people who are doing the most important jobs that we have in the world we're giving them this technology to work with when you know everybody else gets to use cool also in web applications that are like super lightweight and have loads of you X. and U.I. design behind them so I started to think about like how could things be different how might we do things differently. And so long story short I quit and with my friend who I had also worked with in the same place we started a company called Future of which I ran for it's been going for 8 years and for the last 10 years in the last 8 I was helping to run so the 2 of us run this company. It's actually doing better and since I've left so. It's kind of going back constantly hiring So if you fancy working in London then you can definitely apply for a job there but feel should have at the start we were like a social media consulting company because this is Facebook and Twitter was new people were saying to us like people capable of learning about stuff on Twitter how can we shut down Twitter and so we had to be like well it's not really about that and like this is about a new dialogue with your citizens this is about a new way of engaging with the public and you need to embrace it become more porous organization and that was really kind of a turning point for a lot of people but you know our jobs we had been we started out. Working in government understanding kind of the complexity of how government deliver services and really social media was kind of just like a tiny piece of this giant puzzle so we started to kind of like reverse our way back into the organization and think about how might we design. Better systems and processes ways of working and ultimately how might it bring some of this web based technology which is light weight easy to use in the cloud beautifully designed technology to public services so we grew this company there's currently actually there's currently like 80 people working in the company at the time and effort is 40 but we bought another company so. And we worked we were told over the world from United Arab Emirates where they're talking about how do you make they are content with 5 star hotels they have 7 star hotels in Dubai so how do you apply that idea to public services and make 7 star public services over to Australia who are very focused on the transactional elements of government and I spent some time actually in far eastern Europe Russia in Georgia so. That's probably a story for a bar anyway. I was advised to say that we were to have the world so we had lots of time to understand like how different governments and how cities think about technology and innovation then I decided to fulfill a childhood dream not marry a prince but move to America and merge the U.K. and the U.S. in this glorious I just wanted an excuse to show this picture really I think we can all feel it. Yeah I wanted to live in the U.S. And so as a kid I'm sure many people can relate to that feeling so I started applying for jobs and this job came up as the Chief Digital Services Office for the city and county of San Francisco this is now the city hall that I work in. Very grand in comparison but built way after the 1st one her Shoji Wally. And I thought never get a job because like I don't have a digital back I did political science college and I worked in government this whole time and what do I know about design I don't have a design degree but as it turns out I think and I was obviously I was competing against all these people from Silicon Valley so of course like you know I didn't anticipate that I get this job at all but I think what swung it for me and what I what I hear from people who interviewed me is that really their understanding of government and the way government works and just how hard it is to make change happen in government was the kind of thing that I had that maybe other people didn't have and I think there are a lot of kind of technology civic tech utopians who think that like just build it and roll it out they'll be super easy and publics public service will be better and just doesn't work that way and I'm here to tell you why. So what are we even talking about when we say digital services I've got like a few definitions but I think this really speaks to it you know we we are not prepared I showed you the technology the public servants are using it's terrible it's slow it's clunky it's hard to use it doesn't even do the things that you need it to do it's not connected the data is siloed and so we're taking that technology and we're trying to meet the challenges of like autonomous vehicles and facial recognition and all these new technologies that are coming at us fast but we're trying to do that with terrible technology that is really built in the dark ages so that is like one of our biggest challenges so how do we kind of modernize and how do we not just bring in new technology but also bring in the kind of Internet era ways of working that means new business models that means more agile ways of developing software that means design thinking and it means being kind of willing to test and fail as you go it's more about the pros and cons of that. This is my definition my public service is designed around the people that use them because this isn't just about technology right it's not just anyone anytime somebody engages the public service they're not just engaging with the interface at some point they're probably going to have to show up at a building or at some point and inspector is probably going to come to their business and check that doing the thing there's always going to be some offline element with many many public services not always but many and so how does that look at how do we design it as one whole service rather than this kind of fragmented experience. And this is. Some of you may have come across in your studies or his kind of preeminent thinker in the U.K. around digital services and was part of the U.K.'s real efforts to improve government and he talks about employee applying culture practices and processes of the Internet era So again it's not just about the technology it's about the way we build that technology in the way of designing things that is more modern. Because you know things have moved on people people's expectations are much higher Strangely I have found my experience much higher in the U.K. and Europe than in the U.S. So I think you know this slide talks about the pace of life and how we've kind of started to use all these new services that make things so much easier for us but when we ask residents in San Francisco like what they think public services digitally could be like in the next 5 years they're like I don't know maybe just not suck like that's their kind of bar of entry for how public service could be just not terrible please somebody said something like You know I just wanted to be like Starbucks I just want to go in it's not the best coffee in the world but like I'm in and out in 5 minutes I know exactly what I'm getting and I like cool but could we actually have like good coffee is well I'm like really make it you know an extra special experience and then my whole team yelled at me about being a coffee snob and that was the end of that debate because I guess people like stocks so. Government is starting to get in the U.S. I'm sure many of you familiar with the health care dot gov thing that happened so try to make it possible to apply for health care on line the whole thing crash it was a terrible disaster I know some people involved in that project and some of the stories you wouldn't believe but a lot of it because actually we were like contracting out things to smaller and smaller organizations so the line of sight to the user was just almost complete like it was complete a complete barrier there and that was one of many factors. I would say this of course but the U.K. government has really led the way in some of this kind of digital government agenda so about 5 years ago somebody decided that we should have one single website for all of government. And so that's what they built and it's been super successful in the U.K. there are many ins and outs of that and if you're interested I can tell you lots more about that in the Q. and A but suffice to say it's an example of. A government taking a real kind of top down driven change program to actually improve the citizen experience and to put citizens needs 1st are you citizen and resident interchangeably here but just a side note which is that in San Francisco we talk about residents not citizens because we're sanctuary city and not everybody who uses our services is a citizen but they are nonetheless a resident so that's why I'm using those 2 terms interchangeably. The other place we look to is kind of best practice in this field is Boston so Boston dot gov is. A new initiative and I think in the U.S. in terms of cities and city government Boston is really leading the way in some of this thinking and they're there for much further ahead than we are in San Francisco believe it or not. And just kind of think about this in conceptual models for a 2nd you know I think in many organizations not just government but we think about the from the kind of customer facing customer services where the interface that you have with the organization starts with your customer experience and then there's like all this junk in the back office which is like the systems and processes that makes make everything work and all these kind of complexities internal departments and bits of information flying all over the place and. I think people used to think in government about having like a customer service center and then like kind of how you doing all of that back office stuff but the reality is that as a customer we can feel that right we know when some organization is like that doesn't have things lined up properly because we can tell because we get shunted into all these different directions somebody we call one number they tell us to ring another number we come from one website that looks one way and then suddenly we're following a link to something else that looks totally different or like home but that we know when things are aligned within an organization as a customer and I think that it's kind of becoming more and more and more prevalent so we need to think about the customer experience as the entire And not just the the bit that people experience when they call us up or when they come on our website. When we talk about digital services we're really talking about change and transformation and there are kind of 3 levels to that. We see it is kind of like at the bottom level we see these tweaks and improvements like we change the wording on this application form. It's a slight This is a slight snark but I see things like behavioral insights which some of you might have come across so this kind of idea of nudging people to make the right decision I see this is very much at the tweak level like we change some wording we applied some peer pressure and like 50 percent more people paid their taxes mostly number one like I've rarely seen an intervention like that that wasn't just changing the words on the letter so there's that but also really like kind of nudging pitbull 1st he has kind of slightly moral implications that we can get into later but also it's kind of it's not trying fundamentally changing much it's just kind of like improving uptake of an existing service then we have transactional transformation and that's really like enabling you to do something online enabling you to apply for something get something we're putting in Gage with the government and its business processes in a digital way. And that's cool and all but really and it's definitely an improvement on what we have right now so in in San Francisco but it's not really changing anything where it's not fundamentally transforming the government or the organization that people are interacting is just kind of making it easier to engage with this complex system true transformation comes when you start asking for the mental questions why do we have this business process in the 1st place why are we why do we have housing benefit and all these assistance programs that we might give to people to help them access. Help and resources why don't we address systemically the issues that create poverty and how about we start thinking at that level and that's true transformation. What I'm going to kind of try and argue is that like it's kind of impossible to get to that 3rd level until you do some of the thinking around the 1st and Hierarchy of Needs right the kind of the top level is almost like the self actualization part of the Muslim hierarchy but like if your basic problem infrastructure doesn't work then you can kind of forget it. This is kind of saying the same thing but this is a more detailed kind of understanding of what the steps of toward some of the digital transformation going from like entirely paper based process is like right now San Francisco if you want to build something you have to physically walk into a building and get a paper form and fill it out by hand in triplicate and then it will go off to different departments that need it that's what we're talking about when it's a paper process is then we have like you know turn that form into a fillable P.D.F. which you can download an e-mail to somebody it's a lease you can do it on a computer but it's not ideal then we have kind of like transactional change and then after that we think about how do we redesign the service completely so that we're actually you don't have to fill in multiple application forms you just fill in one and it's a it's more efficient and it's helping you get your results quicker. And then how do we rethink the entire nature of services and how they're designed and some of the work we did in features of was much more about service redesign than it was about transactional design so we did some programs and I'll give some examples later but we did some programs thinking about for example the nature of road transport and how we get people who live in rural areas to places they need to go and how do we build different business models to provide that service so very different than thinking about how do we make good web forms so people don't have to come in person. Like I said it's not really about technology like it's very easy to make websites very easy to make forms we all know the technology exists. It's really about the intersection of these different design disciplines and these different ways of thinking and organization design is a big piece of that however like I said if your plumbing doesn't work then really there's no point talking about building a more beautiful house maybe that analogy works I'm sure. This is sort of how we conceptualize the technology infrastructure within the city so we're nowhere near this right now and some are Cisco but this is where we'd like to be you have a web interface which is you know we all know how to make those but we're thinking about services so instead of thinking like get a permit you're thinking build a house right that's the thing that the user is trying to do is to build a house or to open a business or to get married they're not thinking about all the different permits they need to do to get that. To achieve that goal. And then one of the pieces of infrastructure that we need to be in place to enable that So inevitably services are made up of forms and I spoke a bit about forms later but they also need we need somewhere status tracking like where's my application up to in the in this process so we need some technology that does that we need to know you are who you say you are so how do we verify your identity we're probably going to ask you for some money with a government let's be honest so so how do you pay for that thing online to make that a seamless transaction. How do we notify you and tell you when something's changed something's up and so these are all micro services if you like that need to exist. And then in order to be able to do that we need some kind of A.P.I. layer some kind of layer that connects all these systems together and then at the bottom we have alternately like I showed you at the very start what we're talking about connecting into is actually very aging creaking old systems and it would be great if we could completely more completely modernize the core financial system of a city but that is so much work and such a long term project that that would be a complete destruction for our team so what we're focused on is the residence side of things but our job is to be able to connect into those big systems so as you're thinking about building information systems for cities or for any type of government entity you really have to think about not just the interface and making a really nice interface but what is that connecting to and how are you building an A.P.I. into. The say we have a mainframe like into mainframe. How how you how are you going to deal with that kind of thing because you honestly can't fix anything until you're prepared to like roll up your sleeves and do that really hard painful annoying work but someone has to do it right I'm hoping it's all of you in this room. So what do I actually do well I came into this job like very kind of from the very U.K. perspective thinking about the future of public services redesigning transportation redesigning meal delivery for older people social care and then everyone was like Have you looked at our website. So you guys. Have I mean this is kind of what our current website looks like and it's horrible it's really really horrible Not only is it impossible to navigate it just like it doesn't represent the best of what we know that web design can do in this modern era. I was going to show you also the sheriff's department website which I truly believe is the worst we have but I couldn't bring myself to put it on the big screen so instead you have children you have families so here is the thing we have 211 websites in the city like not just one website for the city we have 211 every department has their own website they also have their own brand they also have their own precious logo there some of them have a visual identity they're very attached to others of have the visual identity that somebody is nephew made like 10 years ago they have you know like we are we're kind of in this crazy world where Actually yeah it would be great to like redesign some of the city's business processes but we can't do any of that until we can prove that it is possible to have a single website for the city with a consistent look and feel where our residents aren't totally confused to be amused by this weird time travel that they've entered into to go back 20 years in technology terms as they try and engage with us only to find after clicking on multiple websites that the thing they wanted isn't there in the 1st place. So we are building a new website that is not the job I thought I would have when I came to the scene county of San Francisco I thought some Francisco is the like it's at the forefront of new technology right it's like this is mind blowing place where like all things are possible well. I guess that we can talk about this later I guess that. It's not as simple as that and I'll talk a bit about why we have some of these issues. Hopefully what we can do about it so we're making a website we're putting our services 1st these are the things we know people want to actually do making it possible to transact online it looks really boring one of our users and user testing actually was like is a separate Cisco this is boring it looks like Cincinnati now apologies to anyone who from Cincinnati I've never been there myself but so I say this is not how alternately look but this is our alpha and it's actually life right now so my goal was to have us get something up working as quickly as possible so we could start to prove that it is possible to make technology quickly. And so this is life right now. Other things that we're doing is starting to build out our digital services starting to make it possible to transact online one of the things we've done is create a cannabis service which is. Works here but in California is legal now so it's been legal for the past year so trying to build a way for cannabis businesses to register online is incredibly complex. For reasons that we can go into later but being a enabling people to transact digitally and apply for that permit this is the 1st example that we have that I feel pretty proud about even though I know there's lots of room for improvement Another example is the affordable housing service. So previously this was a paper process so if you wanted to apply for affordable housing or one of our 12 housing programs to fill an application paper application form by hand and turned it in by a deadline many of these So humans being humans they could have like completed it early and mailed it in but of course they all left it to the last moment economy and a group of students could identify with that but you know just try to use your imagination. And so of course to turn in your paper application form it suddenly became this big line all the way around the block as everybody was waiting to turn in their paper. Huge problem so I'll talk about our process for getting to this a bit later but we suffice to say for now we made the service digital and now you can apply for affordable housing online and we've seen now all up online is a huge achievement and you know imagine if you're a single parent you're working multiple jobs and you have to take a day off work to stand in line to hand in a paper form that's what we were asking people on low incomes to do and it's not fair so what what they're now able to do is feel complete online and get the results online as well because previously what actually was previously there was in person lottery. So if you apply for affordable housing in the qualify there is not enough affordable housing in San Francisco as many of you may know. So if you apply for affordable housing you're really just applying to be in the in the affordable housing lottery and there was I swear a legit actual like hand turning lottery thing with tickets inside it was a big event people would show up in numbers to find out where they lucky enough to win in the housing lottery and then they would you know pick up a number of confetti and it was a big thing and we still have that process we still have that luxury but that is digital and it's done by algorithm is completely random. And people are notified because if you didn't show up in person you may miss the fact that you had one in the house in a lottery now police will notify you if you have one in the house in lottery and then you have some time to get back to us and then your application to go forward so vast improvements across the board just by taking with more use assented approach to this is one of the projects that my team. And then we come to forms. And in particular my team is responsible for redesigning the entire permitting process so our goal is to have every single permit the city issues with their board more than 300 permits to have every permit available on line that goes by the stated goal is by 2020 let me tell you now that's never happening so probably by 2021 we'll have it's online. This so this is one of the very I just like I'm kind of a form so I thought this is interesting but if I see you yawning I'll skip over it but this is one of the 1st forms and it was. Issued by the British government. Actually pertains to slavery so it's incredibly sensitive I use it as an example because this is the impact that this has on people's lives the go to the touch point with government forms and so don't ever think that if you are in a project redesigning a form for government that your work is unimportant because you are just designing a form because this changes people's lives. What it does is the idea of a form is that it applies consistency and fairness to a process where asking everyone for the same information but putting them all through the same process forms are incredibly kind of integral within government as an idea of fairness now we can debate whether that's actually true or not and how fair they are but let me tell you forms can also be incredibly unfair and so that's why design and technology is so important because people's lives are at stake with these pieces of information that we're gathering. This is the form that we designed which looks very different as you can tell this is the form that we designed for cannabis businesses. And this is a really really difficult example for me because there are things about this form that I really don't like the design of it it's fine and all. But what we're asking here is are you somebody who should be given the opportunity to jump ahead of the queue in the permit process for a cannabis business so we have all these businesses that were selling cannabis basically illegally for the past 3 years then we made it legal I mean need that being San Francisco these big tech companies are going to come in E.'s and others who like just to be cannabis online is a safe easy kind of cannabis. Thing. And we knew that was going to happen and what we wanted to do was support these kind of like one person to person businesses that had been running for a while give them the opportunity it's a very San Francisco thing to do it's a very kind of honorable thing to do and I think really important for the kind of livelihood of people. To preserve some of the character of San Francisco but what we are what we're asking them to do is to prove to us that they were impacted by the war and drugs in order that we kind of can give them like a jump in the key but we're also we're essentially saying a prove to us that we discriminated against you like really if that trauma by telling us how how we brought that trauma to you in the 1st place to me this is like not the way I would go about designing a policy and this for me is like super important to think about technology design and policy as like. Parts of the same thing because you know we're asking people to give us really sensitive information here which they may or may not feel comfortable giving us why would they trust us we've just we've just discriminated against them why now would they trust us with this information I mean I know it's cool it's cool things have changed trust us but why would they do that and so we have to work Willy hard to build that trust back up with people so this isn't just about designing a form this isn't just about you know did you X. This is really about thinking through people's experiences and their whole context. So just to get a bit kind of contextual about this. My role so how did I get here today from like doing local government stuff in the U.K. to this some Francisco kind of recognized that there was a problem in that like they didn't have this consistent digital experience for people so there was a digital services strategy that was developed about 4 years ago one of the recommendations in that strategy was that they should employ someone to own this problem to own the fact that services needed to be made to digital. This is Naomi She's my boss she's a city administrator and so she she would say she's not a technologist by any stretch of the imagination but I think she cares passionately about is about users and residents having a good experience about meeting people and meeting their needs and meeting them where they are and that's what she cares really passionately about and honestly like I don't need her to understand what an A.P.I. is I need her to care about our residents and that's what she does really really well. And so so she so i Report to the city administrator which I don't know those of you familiar with the structure organizational structure of cities but. That's super senior and so it's great that I get such access to power within the organization because it's very difficult to get influence and leverage in the organization I explained why later. So that's why I sit organizationally and then this is my team which you don't need to really analyze the whole structure but suffice to say I have basically a mix of designers developers and product managers on my team it's a team of $25.00 right now. And they all work on different projects but we have our 3 priorities which is making the website put all the permits online and continue to develop that housing service that I showed you. And so that's what everybody is hopefully working on although I'm out of the office is there probably like tearing it up right now. OK So that's the basics if you like pausing for really like clarifying questions OK so far. So. I'm going to talk a bit about so hopefully from what I've experienced so far you can kind of see how this is less about technology and more about change right because there are so many factors involved in this so I'm going to talk a bit about the conditions for change and how I've seen change happen in these large organizations because I'm assuming that all of you at some point given your the nature of your interest are going to want to enter into careers that are about changing and improving public policy or public services and so these are things that I've seen work. I don't I'm not an academic as you can totally tell so this is like not a fancy framework but this is just the way I think about this so I think these are like the 4 conditions that need to be in place and I'll go through each of them one by one but having a burning platform having That's I think that's a British phrase because I sometimes say that in America people like what on earth he talking about so that means a sense of urgency something that is like really driving the need for this thing to happen and so everybody's laser focused on making it happen sooner accountability accurate problem definition Richard talk about a bit in a bit and then blame management so. Start with this is in the U.K. a public sector colloquially known as the graph of doom I'm not kidding you like the chief executives of cities will refer to it as the graph of team this shows the cost in red and orange you see the cost of public services going up and in blue you see the budget of cities going down there are there is not enough money in the public service cities have had their budgets cut by 30 percent in the last 5 years that's a 3rd of their budget has gone is huge. And so that has really need is only increasing input and this is just I don't social care and children services that doesn't include waste disposal doesn't include street cleaning doesn't include libraries doesn't include any kind of parks or recreational activities this is purely adults and children social care which is seen as like the 2 safety net functions of cities so it's a really serious financial crisis in this sector and so what that has done is really focus the mind on. Digital public services because there are 2 things that cities have at their disposal costs money buildings and people that's pretty much all a city has honestly like buildings and people pensions I guess is another one but I really relates to people so. Once you've sold off all your buildings which many cities have to get these kind of big capital lumps back in to subsidize your work once you've done that you're just left with people and so all you can do is look for layoffs among your workforce and the only way you can make layoffs happen but sustain some level of service is automation right that's the only way that making things digital is the only way that cities can actually manage to make the layoffs that they are required to make in order to meet their financial challenges so digital has become this kind of by word for efficiency and financial savings and so that has you would not believe the focus that there is in the U.K. and Europe I think more broadly on saving money through digital and I can walk into a meeting with a city without being able to clearly say how much money I would be saving with any given digital intervention so that's that's where it is right now. So this is the this is what I talk about by burning platform like we're standing on a platform and it's on fire so we have to do something about that and that is what's creating the impetus for change. The next thing you see in your accountability so through the health care dot gov crisis you can see that. There was a real incentive to get it fixed it was a kind of flagship initiative for the a bomber administration and so from the very top from the president himself that. That emphasis on digital and doing things differently through digital was made very clear this is about my talking to. Megan Smith who's the former C.T.O. for the U.S. who's great should she ever get a chance as he has been you should so to listen to she's awesome. So this senior counsel billeted a senior about investment in making this happen is also this is quite top down but it's a really important factor condition for change to happen the next thing is problem definition so oftentimes and this is I think where your policy thinking really comes into play of the in times where are looking to a solution without framing the problem properly and I think framing problems as one of the kind of forgotten skills of public service is but it's so important because instead of saying how do you make a better application form we might be saying why are we asking for this permit in the 1st place like what is another way of achieving this policy outcome and that's really important to be able to entertain those different options so here's an example of how you might from a problem different in very simple terms away from design a vase which a vase story. Which gives you know some weird and wonderful shapes but essentially the same thing is. To design a way for plants to live in our homes which gives you way more options way more interesting things green walls things that hang from the ceiling and this is a way of kind of asking the same question but in different terms to give you different outcomes back to the very beginning not quite by Madeline Albright like we're kind of using the same framing the same framework for asking questions and for defining problems as we were way back and we need to change that we need to ask different questions to get different answers and that is like really quarter the policymakers kind of tool kit I would argue. The final one is. Blame management so. Runs the government excellent program in Johns Hopkins and she was to I saw her talk and she said this about the culture of government and it's true and people always talk about risk aversion the public sector is so risk averse you hear that all the time it's not it's not risk averse people in public service want to take risks where their 1st 2 is playing their 1st so when it goes wrong what happens is they get blamed they either get holder in front of the media and put on all the headlines for how they messed this thing up politicians lose their position sometimes they can get ousted from mistakes that happen senior managers are often sacrificed you know this thing when this program went wrong we're going to we're going to fire with someone's head must roll and it's going to be the it's going to be the person who you know is easiest to get rid of which is often senior managers and so it makes us naturally cautious like if it goes wrong we're going to get blamed so what you don't have to manage the risk you have to manage the blame you have to manage an environment where it's possible to get things wrong to mess up but to treat it as a learning opportunity not as a an excuse to get somebody fired and that can be really difficult you know the last thing you want to do is kind of like math professors experimenting on our residents because that's like you know some horrific thing at the same time we have to try new things and do things differently in order to make progress and you know that's something that policy is really bad at trying out ideas we don't often prototype policies right we just write them and they just kind of exist and that's horrible and people on the front either trying to implement and they're having all these perverse consequences but what can we do it's the policy instead of like trying to build a framework for prototyping policy where we can get from like some broad policy intention down to says more implementable solutions that we've learned about in the way what that requires is a really mature attitude to risk and by. So those are the kind of conditions I need to exist for a change and then are I'm just going to talk about again through my experience things I've seen models for change there are probably many more out there but these are the main ones that I've seen as I've kind of done my work. So kind of from this goes from bottom up to top down I can figure out how to make the arrow straight that but essentially there are programs in governments that are like low risk low cost pro bono programs we have one of these in some of Cisco's school civic bridge so it can met the tech companies with what we're doing in the city and so departments will say we have a problem that we need help with and then tech companies are consulting firms will come a pro bono help us out for a number of weeks to solve that problem is great low cost is low risk for us therefore blame we didn't spend any money if it went wrong. The downside is it still requires some staff time but you kind of want that level of commitment at least if your staff aren't even willing to give time to them is going to go anywhere anyway this is actually how about housing service that I talked about was developed so the housing department was already to go out and buy this huge giant almost like housing system it's going to cost millions it's going to be like you know very typical I.T. program. Google came in and did one of their kind of super cool designs Sprint's for a week came up with a different way of solving the problem and the department kind of were like their minds were totally blown and they were like We're going to do this all differently and so that that's how they went forward. So that's one way of doing it kind of getting in some pro bono help which is low risk another way of doing it is what. Actually a colleague in the A used to call a trojan mouse so this is a piece of software that you might implement or a policy that you might implement that is again very small kind of a tweak quite low risk but actually serves the purpose of like making people question the fundamentals. If you example. Is good for piloting and that kind of stuff like hey it's just a pilot no big deal nothing to see here don't worry don't get stressed it's just a pilot like actually that pilot can be the catalyst for lots of change the next thing I see a lot of this stuff is innovation labs. Lots of places have innovation labs it's either a physical space and or a team who are charged with bringing design thinking and innovative ways of working into a city. It's pretty cool I'd like to set up a couple of these innovation labs myself in the past I've always been a little skeptical that because if you're not super careful they can be innovation theater like we have been bags we have posted notes innovation but we all know that's not what really makes innovation happen so you have to be super careful about that. And the other thing is you can create this like us and them right it's like all the cool kids in innovation lab Rajar with their jeans and their laptops and like Meanwhile we're over here doing the real work so it becomes a kind of comes like a play room you know and it doesn't become like a place that gets given serious problems or things to organise and it can build resentment right how come you get to wear jeans to work we're not allowed to do that you know it can be like this artifact of innovation can become quite divisive if you're not super careful super inclusive of everybody in the organization. They can work and I think they're a good way of showing what good looks like you know here is an area where like we've kind of fenced off to say the normal rules don't apply here and this is what can happen if you really empower people to make change happen so it can be a good way of like kind of putting a stake in the ground to say this is what good looks like you just have to be it has to be time limited and you have to be really careful about it and then the final way of seeing is like a full team so like you know my team will be an example of that where the leadership is decided to invest heavily in digital and they create an entire team of people who actually doing this work and they become a department within the organization. I expect that my team or my job doesn't exist in the next 5 years and I'm kind of aiming for that because what I would really like is for the departments in the city to be able to do this stuff themselves you know having me as a single point of failure is probably a risk I'm not super comfortable with obviously and it's also not good for the organization right it's not good for it all to be concentrated in one person or one team but everyone should have these digital skills this is the new normal we don't need a team to do this we just so I scour the same kind of getting some of these the problem right so people can actually innovate and use digital getting some kind of guidance and help guides and play some training communities of practice my hope is that within 5 years you know I can start to take a step back from this and allow it to kind of run itself however we choose. I think of kind of talked about most of this stuff. This is a photo of the Government Digital Service in the case so this is what where like this kind of full team idea has been taken to the maximum So one point that the British government was employing 300 developers to make government digital and so that is that's what happens when you have all these conditions for change aligned there was buy in from the top down so government ministers were supporting this this team there was huge investment behind it they had all the skills necessary They even created new job classifications for designers within government because that didn't exist before they had to break the government pay scale to make it happen so that they could recruit the right kind of talent into the organization they also had governance and I won't touch too much on this but suffice to say governance is like critical for these things to happen because they were able to actually stop people from spending money on stupidity projects that were doomed to failure. And to kind of use financial levers to force people to do things and more agile way. So this is all models but like how what are some tactics go through these quite quickly. So one of the most powerful ways I've found to make change happen I think this is true both in policy and in digital and innovation is to start with the user need like go out there and actually speak to the people who are going to be using this service after graphic research is one of the most powerful ways of changing government I have seen this work time and time and time again observing people in their day to day lives not just bring them in for an interview but go where they are follow them around we followed this guy we went stalking him too much we followed this guy on his journey through public transport to understand how do people make decisions in the moment about public transport which then went on to inform how we designed signage on public transit so that people know what connections to make and how to plan that out we also I love this thing here this is like a little detective kit so we asked people with children with autism to do a kind of to be detectives for us and to report on their day and tell us what they loved what they hated what stressed them out will cause them anxiety and what they really enjoy and so we kind of created these kids for kids this is my last job. Up there on the right is we went to. A social club for older people kind of put them all a pint and then sat down and asked them to tell us about their transportation needs so it's not really about having them come in and do an interview where you are it's about going to where they are really deeply understanding their lives once you bring that back to policy colleagues or politicians or other people and really beautifully show illustrate through photography and other things the experience of uses and I've seen people do a complete 180 on their opinions just based on the emotive nature of ethnographic research should always be supported where possible by quantity of data but those things are doing different things right one is about really understanding motivations and experiences and helping you come up with the ideas the other is about help quantity of is about helping you test those ideas assess them work out what your priorities are and they kind of point to in the direction of what further research needs to. The next thing is to just start small like it's great to have big ideas but. Starting with a paper prototype is honestly again like one of the best things people love to go in there with like prototypes that are fully mocked up but I've seen people not really give feedback on those prototypes because they look too finished they look too polished the more scrappy and kind of like rough you can make your prototype the the richer the information you get back about the quality of your product and this could be the same with. You can do this with policy design as well so this is some work that I did with my company and Play Station they actually were in this situation they had perpetrators and victims coming into the same entrance and like really awkwardly waiting in the same reception area to be seen when of crime was being investigated so they wanted to redesign the area so we took all the furniture out and just put it on the street and kind of marked out the walls and kind of reconfigured it multiple times and so we came up with something that felt like it might work and then we kind of tried it out for a few days to see if it made a change like that's a kind of prototyping of both policy and service that can you know they they saved a bunch of money because otherwise they would have just had like a bunch of interior designers coming in and building walls all over the place and this actually like change the flow for them so it doesn't have to be expensive and starting small with it makes it feel much more accessible less threatening to people. And yeah I think I'll just make this like actual policy making being able to change policy kind of iterate on policy as you go is like one of the it's something that's totally missing right now in public service as far as I can see but it is I think the future of policymaking because things are moving so fast and if you policies aren't keeping up things break down. The other thing is that I'm sure you all know this but technology isn't the end in itself and I always feel like kind of super concerned when I come in as the digital officer people are like you're going to make me a website or you going to fix my computer and really I don't care about technology like it's nice I use it in my day to day but it's not the end in itself right it's like the thing that helps us do the thing we're trying to do and I think if you can always think about it in those terms then it doesn't really matter what the systems are underneath what matters is the end result. And then I think it's really important and I'm lucky that I have a team of 2525 is not enough for the scale of the problem that we have ahead of us. Might be recruiting in July. But this is my team or some of my team they've been working with the city actually for many of them for almost 10 years so kind of a long time city employees but I managed to kind of like drag them across under me and I'm really trying to work on coaching mentoring and supporting them to kind of think in this new way and learn these new skills because actually their experience navigating the city is like crucial for this because I don't know all these kind of complex relationships that exist and they do. Nonetheless like you it's great you come in you have some progress brilliant everyone's happy but of course at some point the organizational mean system or you are a virus right you are trying to change the physiology of the organization it will attack he hasn't happened to me yet but I fully anticipate next year gets a major major backlash and so just there's nothing you can do about that that's what organizations are set up to do especially government to preserve the status quo the best you can do is just know it's going to happen and be ready for it and have tactics for dealing with it don't ask me what mine are I haven't figured it out yet. You should other things contextually you need to be aware of it like change goes in boom and bust cycles in organizations and sometimes it does even up to the economy and this capitalist structure that we seem to be stuck with. Not being political just like it seems to be a thing so we get this kind of thing which we're in right now right so I'm probably at the top of this. Cycle where like everything's broken there's like multiple websites now the systems talk to each other everything's the worst. There are typically inevitably leads to we need someone to own this problem and fix it and they need to have a team so we need to have a centralized digital services team that is going to kind of fix all the problems and then they're like Well what we said we want to own you to own this problem we actually comment we just want you to help us because Stop trying to tell us what to do layoff which inevitably leads to like this centralized team has way too much power it needs to be decentralized this should be happening within departments and I've seen this cycle happen over and over again which is why I say that like within 5 years of me starting this role I don't anticipate that my team will exist anymore because that cycle will inevitably come back around and responsibility for this will be devolved again that's OK That's just the natural cycle of organizations and how they work so that's something else to be aware of as you're kind of thinking through policy implications. The other thing is just like I just kind of want to rant about this for a sec. Like if you think that you're not a technology person which I'm hoping like nobody in this room is but like I swear to god there are people who in government where technophobia as a badge of honor they're like I don't do that technology stuff I leave that to my kids and I'm like well you're failing in your job because your job is technology like Don't kid yourself that you can do a job that doesn't involve technology it does every single every single job involves technology so don't go around parading your lack of technology skills something important because if you are anything in an organization in a policy role in a strategy role like you're not separate from technology technology is the way by which we achieve our policy outcomes so this kind of like if you don't feel confident in technology like that's OK but get up to speed Google it find out what an A.P.I. is find out how technology systems work understand how things integrate with each other understand what systems are out there because that is part of your job with you like it or not and the more people kind of insist that it's not their thing the less successful they're going to be as time goes on. So like I said in. This kind of burning platform idea this idea that like we need a sense of urgency behind technology that doesn't really exist in San Francisco right now we have a really strong tax base we are losing any money in actual fact we have lots of tax dollars coming in and we just keep taxing people more and more more for more most of the people think kind of OK with that as well there's not been any kind of citizen revolt it's just sort of happen. But. But one of the things that people really are concerned about is equity usability user experience and so that is kind of the platform that we're using to try and promote changes Imus's go is to kind of say well you know yeah we might not be saving money here that might not be our primary driver but you know what we don't want anyone to be excluded from using this service so we're going to make it the most accessible it can be. We still have services running household waste disposal is still run by city employees that's not the case in many other cities many other cities outsource that jobs are protected you can't talk about layoffs you can't talk about any of the headcount stuff it's heavily unionized city so you need to kind of be mindful of that and choose your battles. I could see as I said it's like one of the chief things and income inequality and housing problems you've probably read about if you have witnessed it yourself firsthand alike some of our most challenging issues they are highly complex they can't be solved by one policy intervention alone. You see this tension between the city city politicians and the tech industry also plays into that Marc Benioff is the C.E.O. of sales force is kind of quite being quite vocal against some of the city's policies and we do have this problem where we have this huge disparity in income equality we're trying our hardest to address that through our work by making sure that everything we're designing is fully inclusive so that doesn't just mean a lot of people think about accessibility as a disability thing but I encourage you to kind of widen your perspective and think about it across the board so not just making sure that our website is 88 compliant and fully accessible to people with screen readers but going beyond that language access we have 4 main spoken languages in the city making sure that it can be. It can be translated into all those languages making sure that our content is no more than 5th grade reading level like if you can't explain it a 5th grader it doesn't go on the website because partly because people are busy who has time to read all that junk partly because it's much easier to translate when it's in simple language and partly because we know that children are accessing our website on behalf of their non English speaking parents so that is also something you really need to bear in mind. In addition to that we're also thinking about income equality so making sure that our code is light that we don't have lots of videos and photos because that eats up people's data plan and inadvertently what we're doing is taxing people who are on a data plan which is invariably people on a low income using our services because we have the vanity to assume that they're going to want to download a massive image of someone smiling like that is not all that is not what a city is there for right it shouldn't be about a website it's not our P.R. machine it's our service delivery mechanism and so trying to make sure that all of our services are fully accessible is in the broadest sense something that we're thinking about. Yes I am almost on just to say that. In Europe or in the U.K. in Europe in particular things are highly centralized in government and so when I came into this job I had to rethink everything I knew about change management because I was very used to kind of quite top drawer are quite top down driven programs of change. I think maybe this is a US thing I think it might be but I see a lot more fragmentation both in the federated system that you operate here in the separation of powers and in this organizational structure which is the structure of the city of San Francisco where you'll see that this isn't even the full thing but there are like $54.00 departments in the city and they're all more or less on a level this is by contrast a U.K. structure this is a chief executive and like a senior team of 7 people who but that's what the piece is worth it's enough for them to be able to get around a table and hash out issues and differences and questions when you have such a large scale organizational hierarchy that is like pretty even power and relationships are like absolutely key in being able to navigate those absolutely key. That means quite often we rely on vendors or nonprofits or other organizations outside of the city to connect the dots within the city I can't tell you how many times a vendor has said to me yeah we're also working with the environment department where you are and then I have to call up the environment department to be like hey you working on this digital thing I'm like yeah yeah yeah but I only know that because 3rd parties have the overview of our organization more than we do. It also means that consensus building is totally a superpower and the people who do best in the city are people who are best building those relationships. And this is really my final thought which I don't know it's not really an academic theory but it's definitely a hunch I have which is that has anyone lived in Europe or the U.K. before spend any time. Couple nods So this is pressure it's a grab and go sandwich place it's pretty straightforward there are like 7 types of Samaj you pick one of the shelf you pay you barely have to speak to a human to have humans there but it's mostly for show I think. And. Also they do exist in New York just to be fair that exists in New York now but for a long time it didn't exist in the U.S. and you say you grab your sandwich so there's like a choice of 7 sandwiches and like if there's something that you don't like in the sandwich because you're British you take that sandwich home and then you pick out all the bits that you don't like and you leave them on the side and then you complain about how they always have too much Mayo in their sandwiches and that's basically how we buy sandwiches in the in the U.S. you have to go to a deli and speak to a stranger and tell a stranger how much Mayo you like in your sandwich it's so weird why would you do that why would you share that information but. Why do you just want to complain about it afterwards. But I think this is like there's something here in the kind of the U.S. psyche about like weak asking for what you want and getting that if that means you have to stand in line and talk to a person then that's something you're prepared to do because it means that at the end of it you get exactly the thing that you want and so I think this maybe speaks a little bit to why a public service is in the U.K. are more digital than they are in the U.S. because I think we will do anything not to speak to a person literally anything including invest millions and millions of pounds in government to make it digital so we never have to speak to a person and tell them about our circumstances I think here in the U.S. people actually I see people you know want to speak to somebody want to have that interaction and maybe it's changing with a different generation I'm not sure but my sense is that that that's something that is kind of there to the U.S. And so I think one of the things that we need to do is just be mindful of that as we're designing services not make them completely without devoid of human interaction because I think I have a feeling that might be really important culturally here. So I'm not really sure what I mean in the bigger picture but that is definitely an observation that I've made. I guess my final point is like to work in the public sector to do this work it's a careers worth of work you won't get it done you won't get anything done in 2 years you belong if you get something done in 5 years for me personally it's been. I still don't feel like I've made that much of a dent in the system so for many people this is their whole careers work so just be sensitive and careful when you kind of trash government and say how bad government is and you know if there are a lot of that today we all just have to really bear in mind that like there are many people working tirelessly day and night to try and make public services better just try to be patient with that I know it's difficult to be resilient if you're entering this field you have to be able to bounce back from some of the failures that will inevitably happen that's just it and so finally my final plug is that like if you have the resilience and the patience and you're interested we probably be recreating from July on my team looking for designers develop and product managers but honestly it's not just about my team like if you're remotely interested or enticed by any of these challenges and problems that I've talked about then please work for government because all cities at all levels only is a polities federal government created need people like to help them better thank you for having me. I. Thank you. He thank you so much we have a bunch of questions here that came in from people in the audience so let's get started and used to the 1st question I have here talks about you mentioned that San Francisco is a sanctuary city so can you talk about any particular challenges with earning trust of undocumented residents and making sure they are accounted for in these digital systems and not overlooked Yeah. Well I think cannabis is one of the areas where it's really come up. As well. We really see a reluctance to engage with services and really see that people don't necessarily want to talk to the government about things we actually as a sanctuary city we're not allowed to ask people for identification documents or a residency documents are not allowed to do that but people don't realize that because like you know it's kind of we're just the government so there's a couple of things one thing is as we're designing the website we have to be really mindful of this opinion in this feeling about government. And try not to make the site to government. And I reference Boston site earlier and when you go in there you'll see that it's quite government nice really nice but definitely a government Web site and we have to walk this really fine line between not looking to government because that's like the symbol of oppression for many people but at the same time being. Official and or 3rd to tive so that people have trust in our services so that's like one minor thing I think what I've learned is that while people don't necessarily trust the government they will trust nonprofits and there to represent and support them so. Well often instead work with advocacy groups. Even with user testing it's pretty hard to get people to come come to the government and agree to give an opinion or sign up for something like that but what we do is we work with those like sector organization nonprofits and have them kind of broker agreement for us and help us with some of that work so that's one way to kind of gather include those experiences thank you we have lots of questions about privacy and privacy concerns So where do privacy protections come into play particularly in today's political climate where there are concerns about sharing sensitive data between different units of government and an example cited was the sensitive information submitted to get a cannabis business to jump ahead in the line and maybe that information shouldn't be considered when applying for a job in a city. Yeah. I wish that we could share data between like I wish we were that organized and our systems are good enough for us to be able to share data honestly and I think there are times when we should be sharing data because we're providing services to people we're asking them to tell us the same information multiple times and it's just like really unfair on people but we our systems just aren't good enough to be able to talk to each other so that's one part of it like I think people think that the government is able to do things with data but they just aren't able to do it certainly at the municipal level I don't know about their the federal level but certainly in this city level we literally can't join up half the data we have even where we should be able to say we can do it so it's one thing. With cannabis Absolutely and. We did some stuff to separate out the personally identifiable information so that my team can see it only the Office of cannabis can see it so. We're cognizant about like who has access to that information how they do that also where it's hosted we make sure that secure and encrypted we also something we've given a lot of thought to subpoenas so for that application form are actually using 3rd party software. So we have a contract with a kind of software developer who helps us make those forms. And. We had to be really sure that when the data was coming in like the feds could actually subpoena the company to get that information a lot of it's publicly accessible criminal history is publicly accessible information so it's not like. They can piece it all together but here we have like this beautifully compiled list of people who have criminal history who have. Been in the cannabis industry for some time. That's basically a hit list for us so we have to be really careful about how do we work with that company to make sure that if they are subpoenaed they come to us 1st before they respond immediately to a subpoena so that our city attorney and our litigators can get in between that process and argue as to why it's not a proper. Response or subpoena so we did think about privacy across the whole level. The city also has just passed some privacy laws. But. Honestly it's extremely political kind of thing. Kind of a lot of political stuff going on there but actually doesn't matter because next year I think later this year actually maybe next year the state has actually implemented this is the other thing about working in city government occasionally the state decides and stuff and you kind of don't really have any choice. And privacy rules have been implemented at the state level which are kind of going to. Impact on how we deal with privacy issues the city level. Next question as part of the Digital Services Program in San Francisco already using or considering using algorithmic decisionmaking systems for public services and there are a couple examples like Montes in L.A. to a stance rest for and house people or in Pennsylvania to assess risk of child abuse. Not to my knowledge I don't know of any that are happening. It's something people talk about but I don't know that point without data. Across the board where we could use it to make those kind of decisions. I think that would be like aspirational as long as we can be sure the algorithms are explainable and transparent and you know. I think there needs to be some kind of governance around which we don't currently have. Creeping queue there are a few questions around the issue of enter net access so does your team discuss free enter net access across possibilities for residents and how do you balance the needs and realities of constituencies that perhaps don't have access to these technologies I access to the Internet. San Francisco access to the Internet is pretty high as you could imagine service so it isn't always that great speeds can be slow and I think that increasingly that's an inequality that we'll end up talking about rather than access or no access eventually it will be about speed of access. We do think about. How I mean the team is digital services right so is there for people who do have access to the Internet. But we have a kind of digital inclusion program which is sort of separate to my team. And that that program works with libraries with community based organizations to help train people in digital list receive and helping them. Access through libraries and other programs. Like P.C. donation programs like I think. We. I think we're definitely taking a mobile the mobile 1st with the desktop kind of like it was still works but we have a mobile experience is what we're going for because we know that even this woman I have a computer you probably have access or a cell phone so that's something we're trying to do and that's another reason why we need to keep our code low maintenance and kind of. Fast and responsive. So next question is how do you and your individual role reconcile the insensitivity is a policy such as the great example you gave about the cannabis with what you're able to do right now. Sometimes it's difficult. I tend to think that we're at least making it better even if we're not fixing it entirely or at least make it better and that by starting small as I said by like by proving that like digital can make a positive impact by then they bring us to collect data by enabling us to go out and talk to users and do user research and testing we can start to bring those experiences back in to policymakers and say look you know we've talked to 10 users this week and they've all told us that this is like overly burdensome or like this is causing them some upset in some way is there any way we can change the policy and start to try and influence from the other direction I think it would be nice to be able to kind of have the latitude to redesign policy and digital because that's really how it should be right but I think we have to start with the inroads that you have in mind is digital so kind of just try and use that as a way of getting in there and then land and expand as they sometimes say something. The next question as you discuss the value of coming to those who are chroma public service background rather than a technical one what do you wish technical experts know about what it takes to work well and bring about change in public services. I think patience and resilience is. Actually saw an article recently where someone from the Electronic Frontier Foundation called public servants lazy and stupid let me tell you that is not the way. To get people from public service to work with technologists and so I think. Kind of understanding how hard it is to make change happen and having some empathy for that is like. Most important I think also just like basics of like understanding how H.R. and procurement work which are 2 of the hardest things to fix in government. It's it's really hard to buy things and it's really hard to hire people and it's really hard to fire people so those are some of the big barriers in there like super real barriers to making change happen so. Understanding that how government actually works you don't have to be a government expert but at least do the courtesy also honestly just google stuff like I can't tell you the number of times a vendor is like let me talk to you about your you know your fiber network needs and I'm like if you Googled me in my role you would see that that's not my responsibility as the responsibility of the I.T. Department which is separate from it like it's so basic but they don't even take the time to do that so I guess if you're coming at it from a technology angle then like I dunno just Google the crap out of the person you're about to talk to us. Yeah I think that's our last question so thank you so much thanks everybody thank you.