Unexpected NA Shard Downtime

The NA shards have experienced an unexpected shard downtime starting at 3:50PM PDT. Trion is working on getting them back up.

Cause is due to some optimization for guilds introduced in 3.3 that borked.

Updates

[3:50pm PDT] – NA shards go down.

[3:53pm PDT] – CM Morgana has made a post stating that Trion is looking into the issue. [r]

[5:02pm PDT] – Cause of outage is due to “optimization changes we made for guilds with the introduction of 3.3”. CM Morgana has stated they think they’ve fixed it, just testing internally. No ETA, but she’ll provide a status update within the next hour. [r]

[5:19pm PDT] – CM Ocho has been responding on Rift subreddit. Issue is with some guild data – unfortunately internal/PTS testing couldn’t account for the full range of ages/states of different guilds and some of them crashed the shards. [r]

[6:05pm PDT] – CM Morgana has stated that they’ve tested out their fix internally and will be updating the shards with the fix. ETA: ~1hr. [r]

Hey all,

We’ve checked our fix for the issue, and are updating the shards with it. Give us another hour or so for the process to complete. Thanks for your patience!

[6:42pm PDT] – CM Morgana has stated that they’ve found another issue that could cause a shard crash and have fixed it, but this means it’ll take “an extra few hours to get that fix tested then applied to the servers”. Next update from CM Morgana will arrive in another 2hrs or so. [r]

Looks like we might be getting something for the downtime:

I know I’ve said it a lot today, but we really do appreciate your patience as we work to resolve all these issues. Keep your eyes peeled for a post from Ocho on what we’re doing to express those thanks.

[Source: CM Morgana]

[6:44pm PDT] – Current EU shards downtime (for their Patch 3.3 update) might take longer than the originally estimated 3hrs due to the shard-crashing fixes noted above. [r]

[7:18pm PDT] – Due to the downtime, Trion will turn on 50% bonus experience, PA exp, favor and prestige until Sunday July 19th at 11:59PM Pacific. [r]

[8:26pm PDT] – Tacitus has confirmed that the downtime will not include fixes to other bugs as that would add additional risk/QA time. The other known issues with Patch 3.3 will be fixed with the next non-emergency patch. [r]

[8:51pm PDT] – CM Morgana has posted that they are getting closer. Fixed on internal test shards; QA starting verification process. CM Morgana will post another update in ~1hr. [r]

[9:45pm PDT] – RIFT 3.3 Hotfix #1 notes are up. Fix to crash + possible fix for minion adventures not granting rewards. Shards are still down. [r]

[9:55pm PDT] – Almost ready to be unlocked. Shards being validated. [r]

[10:19pm PDT] – NA Shards are back up!