“Twitter? Twitter? Twitter? Twitter?” “Um, Twitter’s sick. My best friend’s sister’s boyfriend’s brother’s girlfriend heard from this guy who knows this kid who’s going with the girl who saw Twitter pass out at 31 Flavors last night. I guess it’s pretty serious.” That’s basically where we are with the coverage. Meanwhile: “‘Twitter Is Dead,’ 300 Million People Post On Twitter.”
I like to think of myself as technically literate, moreso back in the day than I am now. Thanks to my Web 1.0 heritage, I can set up and run a website, including the backend. However, the systems that Twitter engineers are (or were) maintaining at scale are unimaginably complex to me. These systems — their design and implementation, who runs them, what they deliver to users — are at the heart of the Twitter story. Unfortunately, the first two aspects of Twitter are opaque just now, so finding out what’s really going on is very difficult (and on this story, Twitter the platform isn’t helping very much). So what’s really going on?
The most basic question: Is Twitter down? The answer as of this writing (12:03PM EDT, November 21, 2022) is no. Twitter is not down. Twitter’s APi page says all systems are green:
Will Twitter go down? Down for Everyone or Just Me? says it has, so doubtless it will:
Interestingly, the last five outages are all since Musk’s ascension to CEO, but that’s as far back as the data I can find goes (and Twitter’s record of being up has been no means perfect. I can say, however, that as an extremely online person who practically lives on Twitter, that I didn’t feel a thing).
In this post, I will look at two aspects of Twitter. First, I’ll look at Twitter’s software, and show why the World Cup will stress it. Then, I’ll look at the players in the ongoing Twitter drama, and the role of each in Twitter’s current drama. I will also add an Appendix on Mastodon, Twitter’s open source competitor.
Twitter Software
The Twitter software handles a lot of data. From the Twitter Engineering Blog, “Next generation data insights using natural language queries“:
At Twitter, we process approximately 400 billion events in real time and generate petabyte (PB)-scale data every day.
That’s a lot. In the face of this data flow, the essential technical challenge to keeping Twitter up — the baseline requirement for Twitter creating value — is spikes. From the same blog, “When seconds really do matter“:
On Twitter, the real world happens in sub-minutely intervals and activity on the platform follows. Whether it’s a retirement announcement from a member of a boy band, a government coup, a world wide soccer tournament goal, or simply the start of a new year, traffic patterns change quickly and unpredictably as these events unfold. These are the moments when the minutely model becomes insufficient. Let’s imagine that a celebrity couple announces the birth of a child at two seconds after the nearest minute — for this example, let’s say 2:00:02. With this announcement, a spike of traffic causes undue pressure on one of the many micro services[1] that powers Twitter causing service errors to some users.
The classical example of this sort of spike is the load on sewage systems when a million toilets simultaneously flush during a commercial break for the Superbowl. For Twitter, the 28 days of the World Cup pose an equivalent challenge. From the New York Times:
[The World Cup] is expected to bring a deluge of traffic to Twitter, which is the world’s fourth most visited website, according to Similarweb, a digital intelligence platform that tracks web traffic. Twitter gets 6.9 billion visits each month, slightly more than Instagram’s 6.4 billion, though far fewer than Google, YouTube or Facebook, according to Similarweb estimates.
In the past, Twitter has a “Command Center” to handle the spikes, but many have left. From The Verge:
Several members of Twitter’s “Command Center” team, a group of engineers that is on call 24/7 and acts as the clearing house for problems internally, also tweeted about their departures. “If they go down, there is no one to call when shit breaks,” said a person familiar with how the team operates.
“A person.” Most of the coverage on internal Twitter issues is single-sourced. Another single source from the Guardian:
Twitter stands a 50% chance of a major outage that could take the site offline during the World Cup, according to a recently departed employee with knowledge of how the company responds to large-scale events.
The former employee, who was granted anonymity because of the sensitivity of what was discussed, has knowledge of the workings of Twitter Command Centre, the platform’s team of troubleshooters who monitor the site for issues such as traffic spikes and data centre outages.
Between the lack of preparations and the lack of staffing, I think it’s going to be a rough World Cup for Twitter,” said the former employee.
He suggested that an incident of some kind – such as a service responding slowly or incorrectly – is almost a certainty during the 29-day [sic] competition in Qatar, estimating a 90% possibility of something going wrong that users would see.
Well, we’ll just have to wait and see. 28 days will be quite a stress test.
From handling spikes, we turn to Twitter’s basic data structure, the graph (or “social graph”). Here’s a very simple diagram of a very small graph:
More technically, once again from Twitter’s blog, “Graph ML at Twitter.” The circles in the image above are “nodes.” The lines between them are “edges”[2]:
Graphs are mathematical abstractions of complex systems of relations and interactions. A graph represents a network of objects (called nodes or vertices) with pairwise connections (edges). Graphs are ubiquitously used in fields as diverse as biology, quantum chemistry, and high-energy physics’. Social networks like Twitter are examples of very large-scale complex graphs where the nodes model users and Tweets, while the edges model interactions such as replies, Retweets, or favs. Public conversations happening on our platform typically generate hundreds of millions of Tweets and Retweets every day. This makes Twitter perhaps one of the largest producers of graph-structured data in the world, second perhaps only to the Large Hadron Collider.
“Second perhaps only to the Large Hadron Collider.” Oof! Since Twitter as a business exists to monetize relationships between users as expressed through replies, Retweets, and Likes, the representational power of the Twitter graph is of the first importance. No doubt that’s why Musk may improve it. From Harvard Business School’s Andy Wu:
Doing that kind of switch is very, very dangerous. So we’re going to have to wait and see what’s going to happen here. My sense is that they are prioritizing the internal core of the technology of Twitter, the social graph.
So, what’s really going on with Twitter? In the short term, an enormous stress test of the software: The World Cup. In the long term, improving Twitter’s social graph.
Twitter Players
But what’s really going on? Let’s look at the players, and the struggles between them. The players fall into to buckets: The oligarchs, and the Professional Managerial Class (PMC). Let’s look at each in turn.
First, the oligarchs. In this section, the set of all oligarchs has only one member: Musk. Software luminary Grady Booch writes in the unrolled Twitter thread: “Twitter 2.0: a modern story about systems engineering“:
Take a legacy organization that has over the years built a not perfect but reasonably solid web-centric system at elastic global scale.
Let that engineering team operate under the demonstrably, poor leadership of @jack and the distinct lack of adult supervision by Twitter’s board for too many years.
Bring in a reluctant CEO with the charisma of Mussolini, the bluster of Trump, and the sensitive nature of a bull in a china shop.
Have that CEO finance the deal with a debt that would stun a team of ox, a debt that has to be serviced on top of that organization’s already negative cash flow.
On day one and for many more days that follow, have that CEO alienate his newly-acquired users, scare away his few advertisers, and decimate the development team.
Have that CEO demand strict allegiance to his capricious edicts, institute unsustainable engineering processes, and tear apart the existing infrastructure by ripping out parts he barely understands.
So here’s the thing:
Making a Twitter 2.0 is only partly a software engineering problem.
It’s a systems engineering problem that requires a firm and steady hand on all the other moving parts such as the business model, understanding the needs of its customers and other stakeholders, defining a compelling vision for the future.
However…
Elon is approaching Twitter 2.0 as if it’s just a “hardcore software engineering” problem.
That’s where he’s gone off the rails.
Twitter 2.0 is literally not rocket science, but Elon is treating it like it is. His focus on the hardcore side of it is understandable yet misguided, and represents a dangerous distraction and opportunity cost.
I have no idea how this will evolve in the coming weeks.
But I do know that it will be uncomfortable to watch, as the lives of Twitter’s staff are upended and as this important global resource is trashed.
This is as good a description of Musk’s situation as I have seen, and from a Master of the Great Runes, too. (In Musk’s defense, he’s clearly thinking about the business model, too; introducing payments to creators — Twitter has many, many artists, writers, performers, etc. — would be a major change to the business model, and a welcome one, too.)
Now, the PMC. For those who came in late, the definition from Barbara and John Ehrenreich (1977):
Salaried mental workers who do not own the means of production and whose major function in the social division of labor may be described broadly as the reproduction of capitalist culture and capitalist class relations.
Certainly programmers; certainly managers. First, the managers. Matt Stoller writes:
While I am no fan of Musk, I know that Twitter was horribly managed before his takeover.
Grady Booch (above) noted the “demonstrably, poor leadership of @jack and the distinct lack of adult supervision by Twitter’s board.” And from HBS’s Andy Wu:
Twitter was actually in very bad shape and didn’t quite have a future anyway. In terms of really tough problems, [Musk] is the kind of CEO you probably need to try out. Twitter is actually a very, very difficult business challenge that nobody else has been able to solve. So at this point, we might need to, like, swing the car around and see what happens.
This is a company that couldn’t deliver an Edit button to users for the entire period of its existence, even though users were clamoring for it. (Mastodon, Twitter’s open source competitor, has an edit button.)
Now the professionals. These can be divided into staff (inside Twitter) and the press (outside, reporting on it). As far as the staff goes, others echo Gooch’s grudging praise (“not perfect but reasonably solid). Andy Wu once more:
The people who built the Twitter product definitely built a pretty resilient infrastructure to survive through all of this. The technical part can operate without a lot of the people.
An example of such resilience: graceful degradation (like when a web page loads a barely formatted version of itself, but still gives you something to read:
A neat illustration of graceful degradation:
• The failing part of the product is isolated from the rest.
• Functionality is degraded but not gone: the indicator is useful without the counts. https://t.co/evYM1acPGz— marius eriksen (@marius) November 12, 2022
The staff did some good engineering, there. (Other examples of the sort of unsung enginering it takes to keep a very large system up and running here, here, and here.) Of course, from the oligarch’s perspective, none of this translates directly into headcount[3],[4]. Musk may figure he can hire back what he needs:
5. Even most of Big Tech’s domain knowledge is commoditized.
How to do search is known.
How to stream video is known.
How to build social graphs is known.
How to do eCommerce is known.People moved around and taught each other. Elon is banking; he can rehire this knowledge.
— Louie Bacaj (@LBacaj) November 18, 2022
Here, for example, is a very happy camper who stayed:
I still work at Twitter.
I used to be in meetings for 4 hours a day discussing A/B tests on font colors.
Now I stayed up the weekend to rewrite core Scala services and implemented a Graph-based ML model to kill bots.
Next is lightning network integration.
Never felt so alive!
— Julian (@JooleanVillella) November 21, 2022
Let’s just hope the Peter Principle doesn’t apply here.
And now the second set of professionals, the press. The key point here is that the press have done what old-school journalists should never do: They’ve become part of the story by doing very little reporting, in “print” or on Twitter, instead engaging in an enormous dogpile of Elon Musk. (I find the press’s pearl-clutching over a set of software professionals moving on to green fields and pastures new, and their silent indifference to, say, railroad worker or nurses, really galling. I also find the implication that Musk, and I suppose Trump, are the only billionaires worthy of censure galling as well.) For now, I will simply note the economic incentives. From the very last sentence of this Times story:
Mr. Musk has increasingly downplayed the role of traditional media over the past few months, citing Twitter as one of the best platforms for the rise in “citizen journalism,” as he put it.
Oh. Can’t have that. Arthur wouldn’t like it.
Conclusion
As usual, Atrios is sensible. “The Bird App Will Be Gone And We Will All Be Free“:
Twitter had two main contributions to the world (well, the US, anyway): It killed the influence of Drudge….
Quite right. I remember when Drudge drove the news cycle. Having Twitter drive the news cycle, for all its many faults, is an enormous improvement.
… and suddenly a bunch of immensely self-important people were confronted with hundreds of people, often deservedly, calling them shitheads every day.
By calling them shitheads, I mean often calling them shitheads backed up with good arguments. And a lot of those arguments were from people of color and other “riff raff” such as people who went to lesser schools than Harvard.
Trivially, Twitter allows me to call Rochelle Walensky a eugenicist to her face (or her intern’s face), and have others see it. To me, that’s what’s really happening at Twitter. I’m very dubious that there’s another platform that enables that. Non-trivially, Twitter allowed aerosol scientists — many of whom are from [shudder] state schools — to call out the droplet dogmatists successfully (“backed up with good arguments”). That’s a great achievement in human terms, in medical terms, in scientific terms, in every way. We will see whether that’s the Twitter Musk preserves (as I know the censorship-mad PMC will not). We will also see how Twitter holds up under the World Cup stress test. Interesting times!
NOTES
[1] These are the microservices Musk calls “bloatware.” Individually, perhaps. In the aggregate, I’m not so sure. For example, if Twitter’s copyright strike system was a microservice, there’s an issue:
Social media users on Twitter can now upload entire movies in threads as the company’s automated copyright enforcement/ takedown system no longer seems to be functional, Forbes reported on Sunday.
A user uploaded the 2006 movie, The Fast and the Furious: Tokyo Drift in two-minute video clippings spread out in a thread of 49 tweets, which when viral on the platform. It wasn’t a one-off event that somehow just got through. Another user uploaded the movie Hackers in a similarly long thread on Saturday, which even got 14,000 likes, Business Insider reported.
Hollywood won’t like that. Nor will FIFA. But I have a suggestion:
Twitter just algorithmically pushed me to a pirated full movie thread when I swiped down, and that is extremely funny
— Faine Greenwood (@faineg) November 20, 2022
I bet the pirated movies wouldn’t go viral if the algo didn’t push them. Musk should turn off the algo! Give people what they want: A timeline of tweets from accounts that they follow, in reverse chronological order. This would remove an enormous amount of bloat.
[2] What I call a “yarn diagram” in conspiracy theory is a graph. The nodes are conspirators; the yarn is relationships. When I say that the yarn in a yarn diagram is “too tight,” I mean that the creator of the graph hasn’t been nuanced enough about relationships (for example, assuming orders are given when shared interest is enough).
[3] Or maybe it does. From former staffer Dan Luu:
Another reason to have in-house expertise in various areas is that they easily pay for themselves, which is a special case of the generic argument that large companies should be larger than most people expect because tiny percentage gains are worth a large amount in absolute dollars. If, in the lifetime of the specialist team like the kernel team, a single person found something that persistently reduced TCO [Total Cost of Ownership] by 0.5%, that would pay for the team in perpetuity, and Twitter’s kernel team has found many such changes. In addition to kernel patches that sometimes have that kind of impact, people will also find configuration issues, etc., that have that kind of impact.
[4] If Musk is right about headcount, and you play the ponies:
Big tech gonna crush earning when Elon shows them they don’t need 2/3 of their employees.
Long GOOGL, AAPL, MSFT, META.
— Nick Huber (@sweatystartup) November 18, 2022
APPENDIX Mastodon
More than a few Twitter users have ventured out into Mastodon, Twitter’s open-source competitor. Naturally, they reported their findings on Twitter.
Mastodon is large by its own standards, but not by Twitter’s:
Mastodon has just passed over 2 million active monthly users, a new record! People are voting with their feet. The future of social media doesn't have to belong to a billionaire, it can be in the hands of its users.
— Mastodon (@joinmastodon) November 20, 2022
There is one Twitter; there are many Mastodon servers (hence jokes about garages). That means onboarding is hard. You have to pick a server, then apply for an account, then wait. Some servers have shut down applications because they can’t handle the load;
https://t.co/yDubzu72Wk has just closed the membership for now. They have about 87K members and don't want their server to crash. That's what https://t.co/oQne7oDMQp has been doing. They weren't set up for an avalanche of new members from twitter
— Adora Krem (@Adorakrem) November 21, 2022
The Mastodon servers are collectively known as the “fediverse” (federated universe):
Okay, so imagine Twitter, but there are a bunch of different insular communities you need approval to join with Byzantine rules about what can be posted
— noam chompers (@NoamChompers) November 21, 2022
The fediverse is not simple:
Mastodon isn’t complicated, but if you’d till need help, here’s a quick explainer for navigating the site: pic.twitter.com/s6o95oroEZ
— the thicc husband & father (@lukeisamazing) November 20, 2022
Sometimes members of the fediverse fight:
Bonus: A lot of servers have also begun to blacklist https://t.co/gwQ3PV2rfH for a bunch of reasons including its size and lack of heavy moderation. If banning whole servers instead of individual people is really the prevailing 🦣cultural norm, i'm not sure that's sustainable.
— Aja Romano (@ajaromano) November 20, 2022
And then there are the admins:
people running to mastodon in light of the twitter meltdowns, please realize the thing killing this website is a web admin with a huge ego who will take his ball and go home
and then remember your mastodon instance is the same situation except the servers are in their garage
— mae (@hyperbolide) November 19, 2022
I’ve set up an account on Mastodon. Whether I move my tiny presence there is an open question…
45 comments
Comments are closed.
no comments at 12:44 PST? seems weird, or ironic
I admit to using Twitter far more than I should, but I am partially handicapped so it is an outlet for me. Up to today I hadn’t noticed much difference on Twitter since Musk took over. There were a few glitches like a list of just 4 topics listed in the “News” section sometimes instead of the usual 20 or so. Today on my personal timeline from only people I follow there, I see very few except for news items from Bloomberg. I do follow Bloomberg for the occasional item worth retweeting, but this is the first time it is about 80% of what I see there. So either most of my tweeps have exited Twitter (few listing “forwarding addresses” on their profile) or else Twitter is having trouble keeping up with increased numbers of tweets about the FIFA championship in Doha. The games will last for about a month, so we’ll see how Twitter fares through it all. In the meantime I’m working on my mastodon account, but it is not very gratifying.
While I do check out interesting Twitter links on here and other sites, I only know 2 people with actual Twitter accounts. Those two people are very liberal and not happy about Musk. No one else I know cares about the soap opera that is Twitter. In fact if Twitter ceased to exist, those are the only 2 people I know that would even notice. On the other hand, almost everyone I know has a Facebook account. If FB and YouTube disappeared, they would go into a panic. Of course, the 2 Twitter people would too. Twitter people seem to spend ALOT of time online on various social media. If social media were like restaurants, Twitter is more higher end, sit down food place. YouTube is Chick-fila, and Facebook is taco bell. Twitter users are upset their fancy sit down restaurant became fast casual.
While the management of Twitter, much like the management of Tesla and SpaceX is a complete *familyblog*-show since musk has come on, it has become much more entertaining.
It should be noted that Elon Musk does not know what he is doing, he wanted to move PayPal to a Windows back end in 2000, (From Linux) and got fired as CEO of PayPal for that, when Microsoft could not get HotMail completely onto Windows until the 2005-2006.
It is likely that Twitter was overstaffed on the tech side, but not at the level commensurate with the losses of personnel.
Of more concern is that the sales staff has been decimated, and advertisers cannot login, and their points of contact are gone, which will have catastrophic results in the long term.
Also, I do not think that a Twitter replacement is even POSSIBLE, because it was created through a unique confluence of events, specifically it achieved early success because pre-iPhone mobile phones could read from and write to the site using SMS, and that is really no longer a thing.
The progressive caucus’ campaign of the immediately retracted letter did strike me as a sort of public submission/humiliation ritual, a display directed toward various controlling elements who might be in a rather desperate state right now and not wishing to be inconvenienced, given the persistent effects of their own incompetence and perfidy. Tonsured Merovingians swearing to never leave the abbey.
The Mastodon server I’m on has defederated a few of the larger servers, but that only means (so far) that if I hit the “federation” feed I don’t see open updates from there. If I follow someone on a defederated server directly, I still see their updates fine.
I’ve come to see all ad-funded social media as inherently pathological, because they (“The Algorithm”) will tend to promote rage-engagement to get those ad impressions. Any social media company not enhancing rage-engagement will lose out to those that do. So, the current migration to Mastodon works for me, because now I can find a lot of the people I was interested in following on twitter in the first place (language, archaeology, some journalism, COVID researchers, etc.), without feeding the rage machine.
That said, Mastodon seems more like the old days of blogging to me, with a somewhat nicer interface than RSS and pingbacks originally provided. The old blogging platforms had terms of service, too. It’s not at all a one-for-one match to Twitter, but does seem to fit the “microblogging” term better.
I dunno. I’ve done this. The world still sucks. Maybe the network-effect makes it effective in the aggregate? But then Walensky’s still employed.
I guess shouting into the ether and being ignored is something that people enjoy. Or maybe I’m the only one that is ignored on Twitter. Either way, I never found any use for it outside of complaining at brands about bad service and (rarely) getting help from a CSR. Seeking acknowledgement that, yes, I’m entirely unimportant in this world is not high on my list. It’s evident enough daily without asking the Twitter.
It’s going to be interesting. I do think there are elements of moral panic about the issue, but nobody is putting words in Musk’s mouth, and he is keeping up a constant stream of statements that are obviously stupid to anybody with even a passing background in enterprise technology.
For example, he dismantled the compliance office and announced that engineers were now able to “self-certify compliance” (my lawyer friend found this one hilarious, and asked if all the engineers would be sent to law school). That might fly with the supine regulators in the US, but the EU is now taking a serious look:
Phrased in soothing and anodyne terms, that says that we will find out whether Twitter is lying to us and also whether they’re doing the things they said they would do. And EU privacy regulations like GDPR have real teeth if you don’t comply. Twitter could face stiff financial penalties or be banned from operating in Europe.
I think it’s quite possible that half of Twitter (or even more than half) was unnecessary. I’m by no means convinced that throwing all of it out (except the code and developers) and selectively reinstating the stuff that turns out to be genuinely essential is a good way to fix it. At best there’s going to be a lot of pain and upheaval along the way (would you fly on an airplane that was maintained according to that philosophy?)
> would you fly on an airplane that was maintained according to that philosophy?
No, but a business is not at all like an airplane (although businesspeople and journalists, who fly a lot, love to make the analogy).
Twitter, as a business, is in trouble and has been for years, as many sources say in the post. Musk needs to put more money in the revenues pocket, and take out less from the costs pocket. Reducing headcount isn’t a bad way to do the latter. I mean, how else would you?
I’ve seen a few of those “day in the life of a software engineer at twitter/facebook/amazon” videos popular on Youtube, and having watched them I was left with the impression that, after arriving at the office at 10am, much of the day is devoted to sipping lattes at nearby cafes, updating socials, enjoying the free lunches in the staff cafeterias, and dashing out by 4pm to meet friends for drinks, which runs counter to the Silicon Valley stereotype of brilliant, hardworking engineers cranking out code and changing the world. Some people have said, which seems plausible judging by these videos, that twitter was an unofficial jobs program for liberals before Musk took over. And one has to wonder what things are like in other departments like e.g. corporate communications when the productivity bar is this low for engineers at what are supposedly tech companies.
I don’t deny any of that (and I think my last paragraph says as much). But I also don’t subscribe to the idea that the only essential workers at a big tech company are the ones who write code, as Musk apparently does. I also question whether he’s had the time to sort the dead wood from the essential ones.
To give just one example, a concern that big companies have and startups don’t is cross-cutting capabilities, like a central customer record. At a big company you have a lot of teams with a need to manage customer information. Individually all of them would prefer to do it themselves, because it’s easiest for them and they can tailor it precisely to their needs. But if that happens, you end up with fragmentation, duplicated data everywhere, and a maintenance nightmare. What happens if you want to implement something global like a customer right under GDPR, and have to make the same change everywhere? At best, you’re stuck with a lot of expensive and inefficient integration or manual processes to do it. At worst, you might not even know all the places in the company where the data is stored. Either will generate more and more work as you get bigger and the number of internal teams grow, and eventually slow your top line growth, blow out your cost line, and prevent you from making changes quickly.
You solve it by nominating a master data store, pushing teams to use it where possible, and establish data mastery relationships and formal update processes for the cases where it isn’t. The coding is the smallest part of that challenge. The bulk of the work is in herding cats and getting teams to sign up to something that, from their narrow view of the world, is suboptimal, time-consuming and taking them away from delivering what their business unit wants.
(Yes, you could create a scalable governance model and set of processes up front and get people to use it from the start, but that’s completely the wrong model for startups, who need to move fast and deliver a lot of functionality quickly, and most big tech companies were startups originally. You can’t get there from here in most cases).
I’m not sure who is working on that and similar problems at Twitter. I think it’s quite possible that the answer is nobody any more, because none of them were writing code and Musk decided they were expendable.
Keep in mind those vids are thinly veiled recruitment pieces. So yeah, they’re made to look like it’s all snacks and beverages and “teamwork!!!”
Micro-services based architecture is the goto approach nowadays for resilient scalable architectures. Instead of constructing a monolithic app, the idea is to construct an eco-system of micro-services, with change management at the micro-service level. Evolution occurs at the micro-service level, so each part of the eco-system evolves “independently”, more or less. Recommend https://www.youtube.com/watch?v=CZ3wIuvmHeM on how Netflix is/was doing it.
A lot of this hinges on container-based (e.g. Kubernettes or Docker) deployment which fits well for cloud-based depolyments (public or private), but can be used on-premise as well. Basically spin up containers to scale to demand and to manage resiliency.
So for Twitter to get through the World Cup, just avoid evolving the micro-services. Their cloud-based container-based infrastructure should take care of the rest, the demand and fail-over responsibility. Or if they need to introduce change, make sure they follow their best practices to roll-back to previous container instances if something goes wrong.
I suspect this type of stuff isn’t the strong suit of their software engineers from Tesla and SpaceX. That said, maybe Tesla has some stuff that needs to scale for connectivity to cars?
Yes, a change freeze seems like a no brainer for the World Cup. I’m having trouble squaring that with all the stories about rewriting core services and the like. Maybe it’s all going into a nice safe little sandbox somewhere, to be deployed to production after the big event is over – or if not, maybe there’s a clear rollback process and reliable dependency tracking so that it could be cleanly reverted if it made trouble.
Musk COULD say something like “We’re going to sort all this out, but we also need the platform to be stable for the World Cup, so we’re leaving it the F alone until after the big event.” He has not said that, to my knowledge.
Thanks for this. Any more information on microservices would be useful; the coverage was basically aghastiude at Musk’s brutal stupidity, but entirely content-free.
In current usage, micro-services refers to the partitioning of systems into subsystems that communicate with each other using HTTP web requests and responses that contain data structured for consumption by computer programs (e.g. JSON) rather than humans (e.g. HTML). The interfaces between the subsystems are called APIs, Application Programmer Interfaces.
The use of web technology (i.e. HTTP) is I think mostly historically contingent (there are lots of alternative ways for the subsystems of large software systems to communicate) and is socially rather more important than it is technically to the spread of the micro-services paradigm.
The key benefit of this approach is organizational. Integrating large complex systems out of subsystems developed by separate teams is difficult, always was. With web APIs we have a basic communications technology that is very simple and everyone in the business understands, including most newcomers. The interesting part is the specific design of the requests and responses available in each API within the system, i.e. the exact application-level protocol that the consumers and producers of the particular micro-service will abide by.
The web part is completely standard, everyone finds easy so there are no distractions from the design of the application-specific aspects that ride on top of it. This has turned out to have greatly helped teams focus on the design of the service their subsystem will provide from the point of view of its consumers, i.e. the application programmers in other teams working on other subsystems. That’s the real win!
In other words, if management dictates REST/JSON API-based micro-services, this enforces a discipline on teams. Often there’s no need to dictate since many programmers seem to like this discipline.
While API is quite a useful term (“Think about the user of your API, Luke!”), the micro in micro-service can be misleading. The service behind an API can be a large and the API may be serving high volumes of traffic. Think of a micro-service as meaning an internal subsystem service.
Another factor in the rapid industry-wide adoption micro-services, I guess, is that the year-on-year performance increase of individual processor cores tailed off years ago. That was an industry-wide constraint that pressed the need for distribution of the overall system’s work.
Remember when tech for scaling public HTML-based web services using things like load-balancers was new? Distribution is splitting up the work that the system has to do to serve external (public) requests and micro-services is using web tech to connect the system-internal distributed subsystems.
I work on a large, very complex software system encompassing thousands of micro services. While this architecture is indeed resilient, in practical terms there are a few risks that still exist and may be applicable to Twitter:
– dependencies – the stack is very very tall – e.g – you compile your software on top of an image which pulls in libraries which in turn pull libraries which … you get the picture. Usually you can punt and solve the problem by staying put on the older version, but sometimes you have no choice . This is where loss of senior engineers who have worked on the system for years will be felt.
– legacy – not all micro services are created equal. Some may be older than others, written by less experienced people or in a technology which was not well chosen. Twitter is old enough and I have 0 doubt that there are skeletons in their closet that were kept hidden by a number of senior engineers who might or might not still be there.
– kubernetes is not magic. It requires a lot of work. You can automate a lot of things but not everything – especially in the realm of security. An expired certificate can wreak a lot of havoc – more so if it affects a crucial component of the system. Don’t get me started on networking.
Disaster happens daily in the world of large software systems and things are kept going by the senior engineers. They are the backstop. Yes – they can be replaced but not overnight. It would take a couple of months (at least) for people to grok what they have inherited and this is with experience.
There is an Elon cult in tech – the kids are in awe of him. I think that he would have no trouble recruiting engineers but the short term risk to system operations at Twitter is high.
The schism between the Muskovites and polite PMC conformists in the tech industry, which is now hard to avoid, has the potential for some entertainment.
Yes, the technical debt incurred at twitter over the years is said to be a not insignificant factor to consider here, and ripping off the duct tape all at once may not be the smartest move, lest the skeletons come rushing out those closets.
Berenson says Musk worries too much about people leaving Twitter. This is the big club the elites are in. Leaving would be exile.
Probably right.
I’m a bit worried about actual numbers. Maybe it’s me, but I can’t believe Kelso has more followers than CNN without bots.
It has been my experience in the past that you are frequently better throwing people off a program that is in trouble than bringing more people on. A lot depends on what kind of environment Musk creates in the near future – one where really good people are just left alone to do their thing without tons of BS meetings.
I’m pulling for twitter or a twitteresque platform. I only regret not being an early adopter. I could be Wint in another life with my Bitcoin fortune…even with the fall of value.
Lambert said: “Trivially, Twitter allows me to call Rochelle Walensky a eugenicist to her face (or her intern’s face), and have others see it. To me, that’s what’s really happening at Twitter.”
Not trivially in 2013 my daughter and her boss were killed by a drunk driver in daylight in a parking lot. Both were on-duty US Fish and Wildlife Service employees. In the days that followed, it was clear USFWS was in a cover-your-fanny mode. They quickly dispatched 2 lawyers and their safety officer to ascertain whether they had liability (which they didn’t). Then a wall of silence was erected and we had no information. The killer was known, but hadn’t been arrested for 2 weeks and was freely roaming. One cannot imagine our distress.
Our other daughter suggested posting our distress to Secretary of the Interor Sally Jewell’s Twitter account. A response was received in less than an hour. The killer was arrested and jailed by state authorities a few days later.
What would have happened if Twitter hadn’t been available to a grieving family? I don’t know whether cabinet members lock out the public now, but in 2013 Twitter worked for us to great benefit to bring justice.
It really worries me that, because I don’t use Twitter and generally find it gross, I lack power that others have, whether it’s Yelp-like power that a lot of people brag about, or something really important like your example. And I’m not just thinking of me. Twitter gives benefits to those who successfully manipulate it.
My sincere condolences for your tragic loss.
I am quite a user of Twitter – primarily for the ‘fintwit’ posts. Its worked perfectly fine for me so far.
I worked in the software biz for over 40 years ( nope not the same year 40 times :-) ). I watched the fun, the passion for it go totally down the drain over the years – the space got crowded with people in it for money, power etc not for it itself – and of course for not doing much work.
The number of positions filled with “architects” of this that other, the number of meetings bloody meetings, with every Tom, Dick Harry ( they got trained by those “keep the billing hours up” vendors like TCS, Accenture etc ) wanting to be there staggered me in the last couple of decades.
So, when Musk tweets about meetings, i.e. keeping them to a minimum, etc, basic elementary stuff, it was great to see. When he asks people to show their code (ergo they CAN code then), even better.
Its so often been true in software that a few people do most of the work and the others, on the “make work”, “gotta look as if I’m doing something” principle actually make the work take so much longer without actually contributing to the actual task at hand. So his firing so many people, and drastically reducing the employee count is again a good thing.
The story has already moved significantly today. Hive Social has replaced Mastodon as the new Twitter du jour. Literally run and maintained by 2 people.
https://techcrunch.com/2022/11/21/twitter-alternative-hive-jumps-to-top-20-on-the-u-s-app-store-after-surge-of-sign-ups
Slightly off topic but probably relevant to Musk and Twitter. Over the weekend, I watched a YouTube video, and the host made a joke/complaint about Twitter’s blue checkmark. He essentially said that despite having hundreds of thousands of followers on YouTube and tens of millions of views, his account wasn’t verified, but someone who writes a dozen articles for some news organizations is verified. While I don’t think he was terribly serious about his lack of a blue checkmark, I think his complaint about how easy it for people in the media to get the preferential treatment was serious.
I think now we’re seeing the elites who got special treatment up in arms, which I find hilarious when reading articles about Twitter, but incredibly annoying when a writer inserts a complaint about people paying for verification in their article. I don’t think most people care too much about what these media people think.
Doesn’t “lightning network integration” mean Twitter is being geared up to handle Bitcoin micropayments?
> Bitcoin micropayments
I would be surprised if such a thing were possible, or more than vaporware. Musk should just take revenge on PayPal by introducing a competitor.
Lightning network seems to be more than just vaporware: https://www.opennode.com/lightning-network/
Whether the Twitter side is vaporware is another question, but Musk apparently intends to do a number of things to generate revenue via Twitter.
Let’s say Twitter 2.0 gives you a crypto wallet, a way to get money into it, and a mechanism to send money to other users, and Twitter takes a commission. I don’t think I’d use it, but many people might, and especially if it became a way to pay for Twitter’s own services (e.g., $8/mo for blue checks).
Not saying this would be viable longer term, but something along these lines may be in the works.
Musk took $500m from Changpeo Zhao, the founder of crypto exchange Binance, as part of the financing for the twitter takeover so I believe sooner or later they’ll dip their toes into this sort of thing.
I only browse a few accounts, some of which do fit the descriptions above of “the players in the ongoing Twitter drama.” Some accounts in other demographics seem justified in thinking that a loss or redirection of moderation that favors the propagation of right wing versions of anti-(Other)-ism will be harmful to them in the real world. It wouldn’t be surprising if twitter’s new oligarch was inclined to the overtly right (as opposed to ostensibly liberal) version of identity politics. At least some of his reinstatements and comments are leaning that way.
Other than that limited view, I have no idea of the numbers and varieties of accounts that make up the national and global universe of users or whether – in themselves – the exit of the msm/pmc/teamblue contingent, or the potentially increased presence of US extreme right content will have an impact on the rest of that universe. There’s a lot of use of twitter beyond blue check pmc’s, even just in the US.
On alternatives, post.news is getting some attention. The founder listed some job postings on…er, twitter.
https://twitter.com/noam/status/1594468317137440768
Twitter seems to be under intentional attack to try to wreck what the attackers have lost i.e. their Twitter. One such attack is where this account uploaded the entirety of “The Fast and the Furious Tokyo Drift” in 2-minute chunks over a 50-tweet thread. It would be damn near unwatchable but obviously the intent would be to drop Twitter into a copyright dispute with the film’s owners. And certainly Musk broke a lot of rice bowls when he took over Twitter as he sorted over who was necessary to run the operational machinery and those that were more akin to clutter who had their own agendas by which I mean censorship and tone-policing. Of course Musk has his own limits and said that he won’t allow Alex Jones back on for his grubby attack on the Sandy Hook parents of dead children.
https://www.forbes.com/sites/paultassi/2022/11/21/twitters-broken-its-copyright-strike-system-users-are-uploading-full-movies/?sh=6d205a3d7cc4
https://twitter.com/elonmusk/status/1594552252865384450
…and one wonders how much money-wrenching ex-tweeps did on their way out the door.
Things have definitely changed though, since Musk rode into town. Formerly, I used to only see a few tweet notifications per week, and now I get a half-dozen each day. Some stops have been pulled out, somewhere.
These are all from peeps that I follow, so at least for now it’s a lot less annoying that the old regime of “our enlightened algos thought this rando tweet should be displayed on your lock screen.”
> Formerly, I used to only see a few tweet notifications per week, and now I get a half-dozen each day. Some stops have been pulled out, somewhere.
I have, of course, turned off notifications (at least OS-level notifications). Also, you can turn off the algo for the timeline, which I have also done. As I point out, Musk could save a bundle by nuking the algo.
As for monkey-wrenching, that’s probably why the code freeze (which also prevents the introduction of new bugs).
I want to hear what the experience is like for advertisers, since they are the real customers and will continue to be an important constituency even as Twitter expands paid products for end users.
Is the ad creation portal still functioning? Have account managers left, too? Are ads still getting approved? Have costs gone up or down?
I think the advertising business at Twitter is bad and has been bad. For example, on my own (tiny) personal account I am obviously a photographer. There are many like me. Have I once seen an ad for photo equipment? No. I chalk this up to terrible management.
From the point of view of brand marketers, you don’t want do have to choose between selling to truthers and selling to those in generic reality. But if generic reality activists force you to choose then I think it’s clear which side most brands will go with. So I guess the question comes down to, where will the generic reality activism happen? That’s a dynamic yet to play out that I assume Musk can understand and how he has more influence over it than most.
No Blue Bird of Happiness for Musk after reading these comments.
Great write up! Thanks as always. I fall squarely into the “#Twiiter is no going anywhere” camp, and all the sh**-lib wailing and gnashing of teeth stems largely from the sense that “their team isn’t running this anymore” as opposed to some tangible reality of bad triumphing over good. A lot of the people complaining had no problem with CTR trolls tone-policing general election discourse on #Twitter, and of course, we can never forget that abomination, #KHive.
The real saucy bit is that that of Elon caught between a rock and hard place having ponied up so much money to buy it with creditors to satisfy. As I see it, he’s got #Starlink at the ready to rake in “defense” money – did anyone else catch the sleight of hand of #SpaceX buying #Starlink ads on #Twitter? (via CNBC) Additionally, don’t forget one of the big shareholders is a Saudi Prince; our frenemies in the Gulf are drowning in US dollars, and they are looking to
use that money to buy influenceinvest in a whole range of things – Formula1, Twitter, Senate elections! So taking a $36B haircut on a $44B purchase might destroy a lesser human, but Elon’s got some aces up his sleeve.Add in the whole “temptation of Trump” thing, and you have the makings of must-doom-scroll Twittering!
Enjoy!
I’m trained in communications skills, not tech. Hence, I am not registered on neither TWITter nor FaceBook. I have no interest in doing so. FB always struck me as too nosy which has proven to be true. TWITter turned me off cos I do not like illiterate or semi-literate writing. If that sounds elitist, so be it. Good read and comment thread.
I can’t help ponder how the use and culture of Twitter shifted over time.
I went and checked, and my account dates back to the early years. Largely thanks to some tech bloggers started to mention and use it to communicate among themselves as they went to CES and like.
I think something shifted around 2015, maybe thanks to a influx of former Tumblr users. Because in 2015 Twitter implemented their retweet with comment feature. Meaning that you made a tweet that included a short url for another tweet, and the Twitter UI would embed the latter within your own.
This is similar to how people commented on other people’s postings on Tumblr, as there you can’t leave a comment below the initial posting. And Tumblr seems to have been a hotbed for certain social sciences back in the day, along with a outlet for various sexual fetishes (the two may well be intertwined).
But come 2013 Tumblr had been sold to Yahoo, and some users didn’t take kindly to that. Thus it may well be that said users migrated to Twitter, and brought their Tumblr “culture” with them.
And that then got mixed in with the journalistic use of Twitter that had been there from the early days, and the combo turned “explosive”.
I do wonder if the lack of overt subcultures in real life these days is because it all takes place online. Different forums and platforms takes on different behaviors and strictures as people come and go.
Elon, blessed be his name, has proved to be a master at generating free PR for his ‘brand,’ which is, of course, himself. Trumpian … but more so, because he actually makes stuff, like cool cars and rockets. Unlike Bezos and Zuckerberg. And those latter two definitely look, well, weird.
So, Elon is a billionaire, makes cool stuff, is totally conversant with how to manipulate government bureaucracies, has almost as much name recognition as Jesus Christ, ‘owns’ a planet-wide media machine, what worlds next to conquer? Yes! POTUS!