On 13th January Linden Lab posted Share What Valentine’s Day Means to You for a Chance to Be Featured. Linden Lab are inviting residents to share a photograph on what Valentine’s Day means to them. There is a great opportunity to be part of the Lab’s advertising campaign for Second Life which sounds great with a month to go until the big day.
Share your story in a picture > Submit to the Official Flickr Page tagged “SLVday2016”
Images must be at least 2048×1207
The chosen image will be showcased in an email to Residents & some banner campaigns
Submissions end on Friday 29th January 2016
See below for more info!
The blog post reads this in full…
Valentine’s Day is just on the horizon and will be here before you can get through a handful of heart candies! We’re looking for some great Second Life pictures from the community to feature in upcoming email and banner campaigns – with credit to the photographer – of course!
In a snapshot, let us know what Valentine’s Day means to you in Second Life.
Share your story in a picture and on our Official Flickr Page with the tag “SLVday2016” so that we can see all of your amazing works. You may submit as many as you like between now and January 29, 2016.
We’ll showcase the chosen image(s) in an email to Residents, as well as in some banner campaigns. We’ll let the chosen image creators know via Flickr and ask for an avatar name for the credits.
Submission Requirements:
Images must be at least 2048×1207
Make sure that your submissions are appropriate for all audiences.
Show your avatar or avatars in a scene inworld – an image that tells a story is going to make an impression.
Keep your image free of additional text/logos
Make sure your lighting is not too dark and that avatars can be seen well
The Grid Is Down While We Bang on Things – used during downtimes many years ago!
On Monday 11th January April Linden who is a member of the Second Life operations team posted a blog post named Why Things Were Less Than Optimal This Past Weekend in Second Life. Due to a “series of independent failures happen that produced the rough waters Residents experienced inworld”.
A master node of one of the central databases crashed on Saturday and this was one of the most used databases in Second Life. The failure caused disruption for a lot of Second Life residents during the weekend. By Sunday evening the operations team managed to re-stabilize the grid back to normal again.
Here is what happened….
On Saturday 9th January
Shortly after midnight Pacific time on January 9th (Saturday) we had the master node of one of the central databases crash. The central database that happened to go down was one the most used databases in Second Life. Without it Residents are unable to log in, or do, well, a lot of important things.
This sort of failure is something my team is good at handling, but it takes time for us to promote a replica up the chain to ultimately become the new master node. While we’re doing this we block logins and close other inworld services to help take the pressure off the newly promoted master node when it starts taking queries. (We reopen the grid slowly, turning on services one at a time, as the database is able to handle it.) The promotion process took about an hour and a half, and the grid returned to normal by 1:30am.
After this promotion took place the grid was stable the rest of the day on Saturday, and that evening.
On Sunday 10th January
That brings us to Sunday morning.
Around 8:00am Pacific on January 10th (Sunday), one of our providers start experiencing issues, which resulted in very poor performance in loading assets inworld. I very quickly got on the phone with them as they tracked down the source of the issue. With my team and the remote team working together we were able to spot the problem, and get it resolved by early afternoon. All of our metrics looked good, and I and my colleagues were able to rez assets inworld just fine. It was at this point that we posted the first “All Clear” on the blog, because it appeared that things were back to normal.
It didn’t take us long to realize that things were about to get interesting again, however.
Shortly after we declared all clear, Residents rushed to return to the grid. (Sunday afternoon is a very busy time inworld, even under normal circumstances!) The rush of Residents returning to Second Life (a lot of whom now had empty caches that needed to be re-filled) at a time when our concurrency is the highest put many other subsystems under several times their normal load.
Rezzing assets was now fine, but we had other issues to figure out. It took us a few more hours after the first all clear for us to be able to stabilize our other services. As some folks noticed, the system that was under the highest load was the one that does what we call “baking” – it’s what makes the texture you see on your avatar – thus we had a large number of Residents that either appeared gray, or as clouds. (It was still trying to get caught up from the asset loading outage earlier!) By Sunday evening we were able to re-stabilize the grid, and Second Life returned to normal for real.
It’s really interesting to hear April’s perspective on what went on and April mentions at the end of the blog post “My team takes the stability of the grid extremely seriously, and no one dislikes downtime more than us”.
One of the things I like about my job is that Second Life is a totally unique and fun environment! (The infrastructure of a virtual world is amazing to me!) This is both good and bad. It’s good because we’re often challenged to come up with a solution to a problem that’s new and unique, but the flip side of this is that sometimes things can break in unexpected ways because we’re doing things that no one else does.
I’m really sorry for how rough things were inworld this weekend. My team takes the stability of the grid extremely seriously, and no one dislikes downtime more than us. Either one of these failures happening independently is bad enough, but having them occur in a series like that is fairly miserable.
See you inworld (after I get some sleep!),
April Linden
I remember the weekly downtimes Second Life had many years ago which lasted for hours and the old message “the grid is down while we bang on things”. Since then the grid stability has improved but things can still go wrong at any time and take everyone by surprise even after 13 years of Second Life being online.
Thanks to April Linden for explaining what happened during the weekend and apologising for the situation.
Daniel Voyager 