Solana Experiencing Issues–Down Over 7 hours: Here’s Why

At press time, the Solana blockchain has not produced a block in over 7 hours. This can be viewed by checking the explorer, Solscan.io/txs. Many are speculating why this is occurring, but it is important to note that Solana is still technically in a beta version of their mainnet.  The network has apparently been down since 8:30 am eastern time, and is still down. Solana has confirmed this on their Solana Status Twitter page, which is run by their foundation. They have chalked it up to “resource exhaustion” which appears to be causing a denial of service saying validators are preparing for a “potential restart” if necessary.

Solana mainnet-beta is experiencing intermittent instability. This began approximately 45 minutes ago, and engineers are investigating the issue.

— Solana Status (@SolanaStatus) September 14, 2021

Just as recently as Sept 2nd, Solana had a similar issue that temporarily “reduced throughput”, but that incident only lasted just over an hour. The sudden freefall in transactions can be seen in the Solana block explorer. Phantom, a wallet built on Solana said in a tweet that “phantom and other applications are having trouble connecting. We will keep you updated on the status of the situation.”

A patch has apparently been made public on Twitter, but it appears the issue is still unresolved. According to reports, the issue stems from a ‘transaction overload” and “has caused an OOM error in most Solana nodes, including validation nodes”. Solana Labs CEO Anatoly Yakovenko said that the issue is being caused by an “overwhelming transaction volume from bots during an initial decentralized exchange offering, (IDO). 

Bots were apparently flooding the network with 300,000 transactions per second. Anatoly mentioned that they just need to “fix some bugs”. As seen below, there is a fix being deployed currently, but no update as to when we can expect it. Some have speculated that it may take 24-48 hours to fix due to “transactions being filtered out, and failing commits, and validators need to upgrade and deploy”.

#solana Fix being deployed in v1.6.23https://t.co/zks8NUC0oy pic.twitter.com/u3IFdlaAvJ

— bruce.codes (@johnma021) September 14, 2021

Many have argued that these types of issues will continue to plague proof-of-stake blockchains. Reasons being difficulty accommodating crashed nodes, and validators being “easy targets”. Moreover, lots of consensus “messages” need to be propagated on the network; many fail to realize that many proof-of-stake supposed transactions per second speeds in their explorers are consensus votes and not real transactions. This has been shown with Tron over and over, as they claim high TPS to boost reputability.

However, Solana’s price has really not suffered much due to this. Currently, it’s only down about 5%, on a day filled with FUD from the SEC and other usual suspects. Solana Status just updated the situation saying that Solana actually forked, over and over, and the process to fix the issue didn’t work and that they must re-start the network. What that means exactly, is hard to say.

1/ Solana Mainnet Beta encountered a large increase in transaction load which peaked at 400,000 TPS. These transactions flooded the transaction processing queue, and lack of prioritization of network-critical messaging caused the network to start forking.

— Solana Status (@SolanaStatus) September 14, 2021

Ari Paul from BlockTower Capital shed light on the situation responding to a tweet stating that blockchains like Solana “are supposed to be Sybil-resistant/spam resistant systems” by stating that “turning off” the network in these instances is a trade-off. However, traditionalists in the space argue that the network should never be deliberately ‘stopped’–indicating such actions are signs of centralization.

Not really – there's a fundamental tradeoff between "liveness" and "safety". Basically bitcoin optimizes to be always on even when network may have problems. Other protocols optimize for "safety", preferring to turn off when consensus unclear. More a function of protocol design

— Ari Paul ⛓️ (@AriDavidPaul) September 14, 2021

Clearly, every chain will experience issues, though. Atom, Cardano, EOS, and more have all experienced issues with block production in the past. And of course, banks experience issues with their legacy systems all the time. The only reason the light is on Solana is because of their enormous successes. I’m sure we can all agree, that we will take minor hiccups like this with crypto, ‘warts and all’, rather than deal with the current corrupt financial system, which is no better and in fact, 1000x worse. Downtime is however a wart that should try and be removed, as the value on these chains is incredibly high, and the space must never sacrifice quality for efficiency.