Synthaholic
Diamond Member
I don't understand a lot of the technical stuff, but it's becoming clear that firing all these engineers was a monumental mistake, which may sink Twitter, Tesla, and his wealth.
4h • 13 tweets • 3 min read
Bookmark Save as PDF My Authors
What I'm hearing from inside Twitter:
Several people who were let go on Friday, then asked to come back were given less than an hour as a deadline.
Software engineers who got this call I know of all said "no" and the only ones who could eventually say "yes" are on visas.
Also:
Many people got a phone call with this "offer", and a short deadline. Lots of people stopped answering unknown numbers to avoid this.
Inside Twitter, managers I hear are getting desperate, trying to call back more people. People are saying "no" + more sr engineers are quitting.
None of this is surprising. As a rule of thumb, after you lay off X% of people, you get an additional half attrition. Lay off 10%: expect another 5% to quit. Lay off 50%... not unreasonable to expect another 25% to quit.
Calling back people you just fired rarely works.
Why it's a problem that senior people are quitting and people don't want to come back:
Twitter has a complex architecture *for a reason.* And it needs some level of institutional knowledge to maintain.
This institutional knowledge both got fired + is walking out the door.
In practical terms: software engineers who are with the company are now put on oncall rotations for systems they have no idea about. I mean, they can figure it out... easiest to talk with someone who knows these.
The problem is when there's no such person left.
Talking with engineers, some things people don't realize about Twitter:
- On prem data centers
- Lots of infra-level advanced stuff. Eg multi-level infra feature flags
- Advanced infra-level incremental rollouts to avoid outages that were caused by infra changes in the past
Unless the institutional knowledge is somehow retained, in days/weeks/months, we should, sadly, expect to see a lot more outages.
The straightforward option to reduce damage is:
1. Retain experienced folks, at least mid-term
2. Hire and onboard new people with these seniors
I know that on Twitter it's fashionable to mock how "slow" Twitter was to ship.
But the more I learn about the internal systems, and why it was built in a way, the more impressed I am. Eg Twitter onboarding to k8s was extremely challenging (+brilliant) thanks to legacy infra.
Twitter has no nuance to discuss Twitter tradeoffs. But as I understand, there were many: some workaround of legacy decisions, some deliberate.
This doesn't change that Twitter is a complex system, and it's complex for good reasons. I really hope enough people stay who know why.
Also, thank you to both people who built these systems Twitter runs on, and especially those staying and maintaining them.
Keeping Twitter running became far more challenging overnight for no fault of ppl doing all this difficult work.
Thanks for keeping the lights on and more!
One thing that continues to bug me:
Elon Musk is an experienced operator and no stranger to layoffs (and their impact). He has a team of advisors from the VC world.
Surely they expected all this to happen. So, why did they do it? Or is this the plan?
Unroll available on Thread Reader
A timely comic from a former Twitter software engineer - several people told me he was one of the most productive web engineers -, who was also Twitter's unofficial Chief Cartoonist.
So a bit more of an insider view:
Worth linking how the author of the above comic got fired at Twitter.
He was working on a high-priority project at 9pm on Tuesday (after Elon bought Twitter). Disconnected and fired mid-work-meeting. No justification as to why.
Now he's suing Twitter.
Gergely Orosz
4h • 13 tweets • 3 min read
Bookmark Save as PDF My Authors
What I'm hearing from inside Twitter:
Several people who were let go on Friday, then asked to come back were given less than an hour as a deadline.
Software engineers who got this call I know of all said "no" and the only ones who could eventually say "yes" are on visas.
Also:
Many people got a phone call with this "offer", and a short deadline. Lots of people stopped answering unknown numbers to avoid this.
Inside Twitter, managers I hear are getting desperate, trying to call back more people. People are saying "no" + more sr engineers are quitting.
None of this is surprising. As a rule of thumb, after you lay off X% of people, you get an additional half attrition. Lay off 10%: expect another 5% to quit. Lay off 50%... not unreasonable to expect another 25% to quit.
Calling back people you just fired rarely works.
Why it's a problem that senior people are quitting and people don't want to come back:
Twitter has a complex architecture *for a reason.* And it needs some level of institutional knowledge to maintain.
This institutional knowledge both got fired + is walking out the door.
In practical terms: software engineers who are with the company are now put on oncall rotations for systems they have no idea about. I mean, they can figure it out... easiest to talk with someone who knows these.
The problem is when there's no such person left.
Talking with engineers, some things people don't realize about Twitter:
- On prem data centers
- Lots of infra-level advanced stuff. Eg multi-level infra feature flags
- Advanced infra-level incremental rollouts to avoid outages that were caused by infra changes in the past
Unless the institutional knowledge is somehow retained, in days/weeks/months, we should, sadly, expect to see a lot more outages.
The straightforward option to reduce damage is:
1. Retain experienced folks, at least mid-term
2. Hire and onboard new people with these seniors
I know that on Twitter it's fashionable to mock how "slow" Twitter was to ship.
But the more I learn about the internal systems, and why it was built in a way, the more impressed I am. Eg Twitter onboarding to k8s was extremely challenging (+brilliant) thanks to legacy infra.
Twitter has no nuance to discuss Twitter tradeoffs. But as I understand, there were many: some workaround of legacy decisions, some deliberate.
This doesn't change that Twitter is a complex system, and it's complex for good reasons. I really hope enough people stay who know why.
Also, thank you to both people who built these systems Twitter runs on, and especially those staying and maintaining them.
Keeping Twitter running became far more challenging overnight for no fault of ppl doing all this difficult work.
Thanks for keeping the lights on and more!
One thing that continues to bug me:
Elon Musk is an experienced operator and no stranger to layoffs (and their impact). He has a team of advisors from the VC world.
Surely they expected all this to happen. So, why did they do it? Or is this the plan?
Unroll available on Thread Reader
A timely comic from a former Twitter software engineer - several people told me he was one of the most productive web engineers -, who was also Twitter's unofficial Chief Cartoonist.
So a bit more of an insider view:
Worth linking how the author of the above comic got fired at Twitter.
He was working on a high-priority project at 9pm on Tuesday (after Elon bought Twitter). Disconnected and fired mid-work-meeting. No justification as to why.
Now he's suing Twitter.
