Scheduled Maintenance 2017/09/29

Discussion in 'Server Chat / Updates' started by Teddi, Sep 22, 2017.

Thread Status:
Not open for further replies.
  1. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    On the 29th of September, 2017 all servers will be taken down for up to 12 hours in the early GMT hours for a sizeable change in our infrastructure. During this time no [BB] services in terms of GMod, ban page or APIs will be available. The forums, Discord, Mumble and Donating* will all continue to be available as per usual.

    The benefits behind these changes allow us to compartmentalise our services better and give us a better degree over incoming network traffic.

    Hopefully these changes and upgrades will take no longer than 12 hours in total to complete.

    *The donation system will be online, but you will not be able to redeem anything until the core [BB] services are back online.
     
    Lightning and Kaiden like this.
  2. Kaiden

    Kaiden Administrator Community Manager

    Joined:
    Feb 13, 2008
    Messages:
    4,826
    Likes Received:
    771
    Woo upgrades.
     
  3. Lightning

    Lightning Active Member MVP

    Joined:
    Jun 20, 2017
    Messages:
    126
    Likes Received:
    52
    Dont want to over step my boundaries but would this have anything to do with the extra ip's or something because of the ddos attacks. I knew you talked about it but if you dont wanna say anything about it idc i just wanted to know if this had anything to do with it or the sizeable change to the infrastructure had to do with what i just said.
     
  4. Kaiden

    Kaiden Administrator Community Manager

    Joined:
    Feb 13, 2008
    Messages:
    4,826
    Likes Received:
    771
    I'm sure Teddi can explain it better, but it's mostly rounding up changes that were started in the summer including better reduancy and better performance in general.

    Sadly DDoSes have always been a hindrance of any server we've ran, and thankfully we got through the summer holidays without experiencing many at all, and only when schools were starting to return did we get hit by a spate (which have since stopped thankfully). On that topic, I expect someone was paying hefty amounts to launch gigabyte attacks against us, which isn't feesible to maintain forever.
     
    Lightning likes this.
  5. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    It's primarily to do with our own internal security and redundancy really. At the moment the OS we're running; whilst isn't out of date will hit EOL (end of life) by 2022 and is Windows based. Now whilst Windows Server isn't particuarly bad to run, it has some certain problems that make it quite annoying to run at times; including but not limited to Core 0 dependancies, lack of (easy) SSH authentication and lack of proper / decent firewall management. Instead we'll be moving over to a *nix based OS and we'll be virtualising the servers instead as opposed to making them run directly on the bare metal. It gives us the best of both worlds (*nix server management) while still maintaining a Windows environment which we can make image backups of on a regular basis.

    Adding extra IPs wouldn't require a complete reinstallation, however with going the *nix route it does allow us to use NAT internally meaning that if the primary Surf server IP is unavailable, we can broadcast it without a Surf reboot across as many IPs as we so desire.
     
    Lightning and Kaiden like this.
  6. Lightning

    Lightning Active Member MVP

    Joined:
    Jun 20, 2017
    Messages:
    126
    Likes Received:
    52
    Thank you for that info. I just wanted to know if that had anything to do with it. Thank you guys!
     
  7. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Due to unforeseen consequences there have been substantial delays. There is currently no ETA as of yet as to when the servers will be available.
     
  8. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Bare metal installation alongside security tightening is now complete, so overall we're about 60% of the way there.
     
    Lightning likes this.
  9. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Current status is as follows: IPs are now correctly set up and broadcasting on all correct ports. What's left is some container set-up and then putting all the gameservers back together.

    Apologies for the hefty delay in all of this, when it turns out the datacentre cannot guarantee a stable connection to their collection of .isos, it makes for a very rough, bad time.
     
    Lightning likes this.
  10. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Current progress is the Windows ISO has been uploaded and is in the process of being installed.
     
  11. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    We're all installed and the networking has been redone to unfuck it. Shouldnt be too long before we're back up and running.
     
    Lightning and Kaiden like this.
  12. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    All user data has been succesfully restored. Reinstallation of game servers is now inbound.
     
  13. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Currently pending some final licensing requirements from one of the vendors we use and we should be live.
     
  14. Carl

    Carl New Member

    Joined:
    Sep 30, 2017
    Messages:
    8
    Likes Received:
    1
    Oh ok
     
  15. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Server is now available for some preliminary testing. The license owners are taken so long so I'm putting up some temporary stuff to get the server online.
     
  16. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Hard server is now available on a more permanent basis, using all the previous systems.

    Systems that are currently not working correctly

    - Lottery
    - Crate resetting
    - CSC

    Please note that crates will continue to drop as normal until you hit your maximum, and the system to reset that isn't yet active again. However that should be sorted by tomorrow.

    -----
    Easy Server is back up and running, but if it goes down it may stay down until it's back live on the standard systems.
     
    Lightning likes this.
  17. Killermon

    Killermon Hammer time!

    Joined:
    Jun 6, 2008
    Messages:
    950
    Likes Received:
    267
    Please note that although most things are up and working, there's still a few creases to iron out. Note that if you have any funny characters in your name (e.g. Emoji's and other weird characters), then your items...etc won't load; if this happens, remove the characters from your name and rejoin the server.
     
    Lightning likes this.
  18. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    Crates are now resetting as intended.
     
    Lightning likes this.
  19. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    CSC and lottery are now both back. Congrats to NeverReady for winning the lottery with 20k cubes!
     
    NeverReady likes this.
  20. Teddi

    Teddi Well-Known Member Bear

    Joined:
    Jul 21, 2007
    Messages:
    9,627
    Likes Received:
    1,153
    BBuild is now back online, although it is missing Wire.
     
Thread Status:
Not open for further replies.