16:00:12 <cohosh> #startmeeting tor anti-censorship meeting
16:00:12 <MeetBot> Meeting started Thu Apr 22 16:00:12 2021 UTC.  The chair is cohosh. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:12 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:22 <agix> hi
16:00:22 <cohosh> hi everyone
16:00:40 <cohosh> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
16:01:15 <gaba> hi
16:02:10 <cohosh> dcf1: looks like that first announcement is yours
16:02:42 <dcf1> The security audit I talked about a few weeks ago is over. There are a few discovered issues affecting Snowflake, but nothing too major.
16:03:24 <dcf1> I've sent the full report to cohosh privately. Some of the issues affected software dependencies, so right now I'm in the process of reporting those upstream.
16:03:35 <dcf1> When that's done I'll make the full report public.
16:04:09 <cohosh> nice \o/ thanks dcf1 for filing those issues
16:04:56 <cohosh> i see one of them also has a fix already
16:05:29 <dcf1> yeah but I think snowflake!31 should be merged first, since they affect the same code
16:05:48 <dcf1> which I only realized yesterday while reviewing !31
16:06:01 <arma2> nice
16:06:15 <cohosh> cool, i read your review this morning thanks for that
16:07:30 <cohosh> i'm impressed with how thorough the security audit was but i also don't have much experience with them
16:09:09 <cohosh> okay the next discussion item is for the Tor Browser 10.5 stable release
16:09:15 <cohosh> it is looming
16:09:35 <cohosh> and the browser team would like us to make a decision on whether it's time to put snowflake in stable
16:10:14 <cohosh> a major consideration the last time we brought this up is that putting it stable will increase the users by a lot
16:10:50 <cohosh> and perhaps things will break that we didn't notice before because our load wasn't high enough to break them
16:11:07 <cohosh> this was part of the motivation for snowflake#40007
16:11:52 <cohosh> here's some usage metrics for our snowflake bridge: https://metrics.torproject.org/rs.html#search/flakey
16:12:40 <cohosh> and usage over the last 3 months: https://metrics.torproject.org/userstats-bridge-transport.html?start=2021-01-22&end=2021-04-22&transport=snowflake
16:13:02 <cohosh> it has increased by 2 or 3 times what it was the last time we had this discussion
16:13:08 <dcf1> Here's what it looked like when meek moved from alpha to stable (2014-10-15)
16:13:11 <dcf1> https://metrics.torproject.org/userstats-bridge-transport.html?start=2014-07-01&end=2014-12-31&transport=meek
16:13:51 <cohosh> ooo nice to have that to compare against
16:14:21 <cohosh> snowflake has more moving parts than meek for sure
16:15:14 <cohosh> otoh, it is at a place where it's useful for some users and we might lose meek soon
16:15:41 <dcf1> at some point we will have to take the plunge
16:15:54 <dcf1> what are we most worried about in terms of capacity?
16:16:16 <dcf1> is it number of volunteer proxies, bridge bandwidth, anything we can anticipate?
16:16:33 <cohosh> we're definitely short on unrestricted proxies (proxies for users with symmetric NATs)
16:16:54 <arma2> can we get some more of those by doing an 'install the goflake' campaign?
16:17:07 <arma2> (sorry, i hope i didn't just rename it ;)
16:17:07 <cohosh> yeah, probably
16:17:10 <dcf1> could that be caused by probetest malfunctioning (snowflake#40039)
16:17:18 <cohosh> lol arma2 too late now :P
16:17:25 <cohosh> dcf1: no i don't think so
16:17:46 <cohosh> i looked at our metrics and they haven't dropped that much during the malfuncion
16:18:02 <cohosh> i also merged some code a bit ago to not have proxies forget their unrestricted status just because probetest fails
16:18:36 <arlolra> is there only one meek bridge?
16:18:40 <cohosh> so proxies will stay unrestricted until probetest is back online and tells them they aren't or enough restricted clients fail to open a datachannel connection
16:19:27 <dcf1> arlolra: yes, there's just one. There used to be one bridge per CDN (meek-google, meek-amazon, meek-azure).
16:19:51 <arlolra> how does the hardware for that bridge compare to snowflake's?
16:20:19 <cohosh> here's the meek bridge: https://metrics.torproject.org/rs.html#details/8F4541EEE3F2306B7B9FEF1795EC302F6B84DAE8
16:20:57 <cohosh> hm, i don't have access to it and I'm not sure of the specs
16:20:59 <dcf1> I don't know. Team Cymru runs it and I don't know the specs. But for a while I ran one of the bridges on an unexceptional VPS and it wasn't a problem.
16:21:32 <cohosh> i'm more worried about the broker than the bridge
16:22:05 <cohosh> since the last time i checked it looked like the broker was getting a lot more requests than i'd expect from the usage counts
16:22:41 <arma2> requests, like from censored users? meaning more people are *trying* to run snowflake than *succeeding* at running it?
16:22:45 <dcf1> I'm wondering if there are some client deployments out there that use -max 3 or something, requesting a lot of proxies that they don't use.
16:23:16 <cohosh> yeah we also have that open issue where our standalone proxies showing a lot of extremely short connections
16:23:24 <cohosh> maybe this is an enumeration attempt
16:23:32 <dcf1> https://lists.torproject.org/pipermail/tor-dev/2014-September/007482.html [tor-dev] Call for a big fast bridge (to be the meek backend)
16:23:43 <cohosh> but it would also be explained by -max 3
16:24:44 <arma2> you can always count on some research group doing the enumeration attempt at getting an ndss paper out of "you didn't do ticket 40055 yet". extra sad because when you see an attempt there's a good chance it's a grad student, not an actual censor. :/
16:24:59 <cohosh> fwiw team cymru was not particularly responsive to use asking them to update software in the last two years
16:25:07 <cohosh> *to us
16:25:17 <cohosh> but we had no other complaints afaik
16:25:30 <dcf1> I guess the question is, if Snowflake isn't in 10.5, do we have an idea of what concrete work would put it in a good position for the release after that
16:25:40 <arma2> (if we wanted to find a new snowflake backend bridge i bet we'd have some options. but also there's the "if it ain't broke" reason for keeping it)
16:25:59 <cohosh> i mean there's a lot of improvements i want to make but i'm leaning towards just going for it and seeing what happens
16:26:05 <arma2> i vote for 'yes put snowflake in 10.5 stable', on the 'if not now when' argument.
16:26:16 <arma2> unless there's an actual answer to the 'if not now when' question :)
16:26:20 <dcf1> If we don't have an idea of what work would change the situation, might as well release sooner than later
16:26:23 <cohosh> yeah same
16:26:37 <cohosh> cool let's do it!
16:26:44 <arma2> and then take this list of things we just brought up, and make sure we're in a position to watch for whether they go too wrong
16:27:13 <cohosh> i think i'll clear time for when 10.5 ships to watch real time snowflake metrics
16:27:31 <sysrqb> cohosh: (sorry, lurking :) can the broker be scaled up relatively quickly? or is it a fixed resource?
16:27:34 <cohosh> we should have some more alerts set up by then too which is good
16:28:37 <dcf1> sysrqb: it's addressed by TLS and DNS name, so we can replace the hardware with a more powerful machine easily, but it would not be so easy if it got so big we had to distribute it across multiple machines
16:28:44 <gaba> i wonder if it makes sense to wait until TB 11 so we have a little more capacity to deal with fires.
16:29:04 <sysrqb> dcf1: great, that was the answer I was hoping for. thank
16:29:05 <cohosh> i'd rather not wait for that reason
16:29:05 <sysrqb> s
16:29:36 <cohosh> (re capacity)
16:29:52 <cohosh> because i'm theoretically supposed to be funded to work on snowflake
16:29:56 <gaba> ok
16:29:58 <cohosh> and this is in scope for that
16:30:18 <arma2> and what better way to find out what's next on the list of bugs than having users find them for us :)
16:30:41 <cohosh> yup
16:32:43 <arma2> cohosh: did that answer your question? anything more you want help brainstorming about how to monitor or how to predict?
16:33:03 <arlolra> with these hesitations, are we assuming a soft launch?
16:33:21 <cohosh> yeah i think we should ship it
16:33:34 <cohosh> arlolra: what do you mean by a soft launch?
16:33:46 <arlolra> ship it but don't yell about having done so
16:34:30 <cohosh> hmm, maybe that's a good idea
16:35:19 <arma2> i could go either way on that one. good idea because the build-up might be more gradual, bad idea because then you don't know how the build-up is going ("did it happen yet?")
16:36:05 <cohosh> It will be one of the default bridge options
16:36:35 <cohosh> and we announce it on the release post
16:37:32 <sysrqb> i have some doubt that announcing it on the blog will make a large difference for usage on censored networks
16:37:42 <cohosh> yeah
16:37:54 <sysrqb> like, users in China will know about it and use it, or not
16:37:57 <cohosh> so maybe we just wait a bit to let the comms team do their thing outside of that
16:38:05 <sysrqb> but i assume training and word of mouth will be more mportant than the blog
16:38:19 <cohosh> yeah okay i like the soft launch idea
16:38:25 <arma2> right, i was thinking something similar: the build-up in china will happen suddenly when the right person qq's the right other person
16:39:06 <arma2> (or whatever it is the cool kids in china do these days)
16:39:33 <arma2> i'm optimistic about the role of snowflake in e.g. turkey too
16:40:13 <cohosh> yeah I'm excited about this
16:42:02 <cohosh> okay cool, so sysrqb i'll talk to you closer to the release date about any last patches we need
16:42:12 <cohosh> and figure out when we need to get those in by
16:42:51 <cohosh> any more discussion before we assign reviews?
16:43:47 * cohosh looks at needs help with
16:44:14 <dcf1> i'll do another round of snowflake!32
16:44:23 <cohosh> thanks dcf1
16:44:37 <cohosh> i wrote a shim for moat in go for bridgedb#32276
16:44:46 <cohosh> it takes a lot of code from goptlib
16:45:30 <cohosh> because it's basically the other half of it
16:45:48 <cohosh> but it's not high priority for a review and can wait a bit
16:46:04 <dcf1> ok I probably will not get to it this week
16:46:13 <cohosh> yeah no worries
16:46:44 <cohosh> arma2: if you have time to take a look at censorship-analysis#40002 and see if there's anything it's missing that'd be cool :)
16:47:10 <cohosh> not sure how involved you want to be in the analysis stuff
16:47:13 <arma2> on my list! (now)
16:47:40 <arma2> i probably have a different perspective from other people, because of how many times i've done this, so yeah if i'm around it makes sense
16:47:55 <cohosh> yay there's a link to writeup in a comment i tagged you in
16:49:33 <cohosh> that's it from me
16:49:45 <cohosh> anything else for today?
16:51:02 <arma2> are there good instructions for how volunteers can set up a goflake? we should get them a home on community.tpo
16:51:20 <cohosh> they could probably be better
16:51:42 <arma2> and they could be linked from snowflake.tpo too now that i'm looking
16:51:48 <cohosh> that's a good idea, and while we're at it making sure the logs are useful enough
16:52:11 <dcf1> Jacobo Nájera has been doing it independently and asking questions
16:52:15 <dcf1> https://lists.torproject.org/pipermail/anti-censorship-team/2021-March/000141.html
16:52:24 <dcf1> https://lists.torproject.org/pipermail/anti-censorship-team/2021-April/000165.html
16:52:44 <cohosh> yeah that has been great
16:53:04 <cohosh> this is where our wiki lives now with the current instructions: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/wikis/home#how-to-run-a-snowflake-proxy
16:53:09 <cohosh> but this is hard to find
16:53:54 <arma2> https://snowflake.torproject.org/#extension next to 'install in firefox' and 'install in chrome' could be a good spot
16:53:58 <cohosh> we also have a docker container
16:54:02 <emmapeel> oh. that sounds like a nice task for the outreachies
16:54:46 <cohosh> emmapeel: oh nice, i like that suggestion
16:55:11 <cohosh> i'll create a ticket after the meeting ends and include the points we just brought up
16:55:18 <emmapeel> i will open a ticket , but maybe gus has an idea of where to go. community.tpo sounds good tho. ill open it there
16:55:26 <dcf1> There's also an outdated version of the instructions (copied from Trac) that no one on this team has edit permissions for, I think
16:55:27 <emmapeel> ok then, less work for me :D
16:55:29 <dcf1> https://gitlab.torproject.org/legacy/trac/-/wikis/doc/Snowflake/
16:55:49 <dcf1> It would be nice to delete it or replace it with a link to the current page
16:55:55 <cohosh> yeah :/ the gitlab wiki situation isn't great
16:56:02 <arlolra> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/19409 might be easier now that libwebrtc is not in the mix
16:56:23 <cohosh> i've had it on my todo list to ask gaba or anarcat who has permissions for the legacy wiki pages
16:56:41 <cohosh> arlolra: ah actually i don't think yet :/
16:56:58 <cohosh> i started this process after we'd switched to the go version
16:57:02 <emmapeel> i just found this https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40031
16:57:19 <cohosh> and since you have to package each go library dependency individually it is very difficult to do
16:57:27 <cohosh> and to update when we bump the version of webrtc
16:57:29 <arlolra> i see, ok
16:58:07 <cohosh> emmapeel: ah yay it already exists :)
16:58:30 <cohosh> i'll just add to that ticket then
16:59:43 <cohosh> okay i think i'll end the meeting here since we're at the hour mark
16:59:57 <emmapeel> cohosh: yeah. i am not sure that support.tpo is the best place for these instructions tho...
17:00:21 <arma2> emmapeel: yeah, community.tpo is a better pick now that it exists
17:00:25 <cohosh> emmapeel: that's fair, we can update the ticket title
17:00:36 <arma2> and actually, snowflake.tpo could be the right place for the instructions, and then community.tpo could point to them. several options.
17:01:03 <arma2> (or vice versa. just don't try to maintain two copies and it will work out :)
17:02:01 <cohosh> yeah let's continue discussion on the ticket
17:02:14 <cohosh> #endmeeting