in this post we will give you a breakdown of the current server issues and how we are going to address them.
In general, the underlying reason for the server issues is that the number of players we are seeing since launch has far far exceeded all of our forecasts. To give you an idea, our number of online users since launch has often exceeded those of EVE Online. This came as a total surprise to us, in particular taking into account that in Eve a part of the online users are afk and another part is on free 2 play accounts.
Due to being a single shard game, meaning that everybody is part of the same game world, we face some unique challenges to adjust to the new situation. Having said that, we are very confident that all of the issues we currently have can be fixed, and it won't be long before we see significant improvements.
1) Specific zones being overcrowded
Problem: There is a hard technical limit to how many players can be in a single zone, and this limit does not depend on the type of hardware used. It has to do with the fact that the information the server has to processes grows exponentially with player numbers. Many of you will know the wheat and chessboard problem, it's similar to that. This can not be overcome by better hardware (we already use the best available) or more servers, as the amount of information stays the same. (to use a poor analogy: if you traveled back in time 20 years with a copy of Overwatch, you could not run it, even if you used the combined power of 1.000 PCs from back then)
Solution: The short term solution is splitting overcrowded areas into multiple smaller ones, similar to how we have put the market and bank in Caerleon into their own mini-zones. The mid term solution is to have an instancing system for the market place of towns (i.e. only the center, and only for NPC buildings there) while leaving all the player made buildings in cities strictly non-instanced. In essence, the end result would be a much cleaner, better looking and more scalable overcrowded mode.
2) Lag in large scale fights
Problem: There is a hard technical limit to how many players can take part in a single fight without lag happening, in particular given that Albion's combat system is quite action based, with lots of things happening and spells being cast at the same time. No game in the world can exceed this limit, and how high it is largely depends on how fast paced and varied your combat is.
Solution: Our combat performance in itself has not yet hit the hard technical limit. Through optimizations, we expect to push up the number of players who can take part in a fight without serious lag. Having said that, such a limit will always exist, again, having to do with the fact that information to be processed grows exponentially with player numbers. The solution to this problem is a strategy that we are already using successfully: we need to make sure that there are enough valid PvP targets at any point of time such that it is not economically worth it to send all your players to 1 spot (as they'd have to split the reward between them). In particular, this means that PvP objectives such as castles, chests, world bosses, etc will happen more often, and the times at which they happen will be more synchronized across the different Outland and royal regions.
3) Server reboots
Problem: Our main database is based on Apache Cassandra, designed to handle tons and tons of data in real time. It generally runs smoothly, and even at peak hours, only uses around 20% of its capacity. However, in irregular intervals, we are seeing spikes that last for a few minutes, and that then cause the current blackscreen and de-sync issues between individual zones and the main database, forcing us to reboot the servers.
Solution: We have now established a direct contact with Apache Cassandra developers in order to identify and fix the current database issue. In principle, the database is more than able to cope, we just have to find out what's causing the current problems on a technical level.
Many of you are rightfully upset and angry about the server issues post-launch. We would like to apologize for this and can assure you that everybody in our team is fully dedicated to fix them as soon as possible - our consumption of coffee certainly has increased by a factor of 10, and we have already signed up our lead programmers for the cast of the next season of The Walking Dead as they'd be able to play the zombie roles without needing any make-up. Joking aside: Everybody who has played since release will be compensated with an additional 7 days of premium time. In the unlikely case that we do not improve the server situation over the next 2 weeks, additional compensation will be awarded.
Thanks for your patience and ongoing support!
Truly a blessing to have such a good group working on ultima online 2017 aka Albion Online... UO is dead, long live AO!