TPU's WCG/BOINC Team

[Deelron]

@Heaven7 - check your event log and see what it says when you try to update the rig

** EDIT **

Checked my WCG account- received only a single wu in the last 10 hrs!!!

Same deal here, no new tasks for me from more then 7 hours ago, according to the event log.

 

[Heaven7]

Click on Update on the Projects tab....

It says "Communication deferred", 2 minutes to wait. OK, now it says "Scheduler request pending", then "Communication deferred" again. Tasks are still running

 

[Norton]

It says "Communication deferred", 2 minutes to wait. OK, now it says "Scheduler request pending", then "Communication deferred" again. Tasks are still running

Let the timer run out (2 minutes iirc) and update again, then check the event log.

 

[Heaven7]

Aha! Looks like there IS something wrong now:

 

[manofthem]

After the OET and UGM dried out, I switched to FAAH earlier today and filled up on WUs, but now the buffers are drying up again.

 

[Heaven7]

What's with the "Tasks are committed to other platforms" message?
EDIT: There are still about 30 WU's ready to start (MCM and faah only), I'll check back in the morning. Hope things get fixed soon...

 

[thebluebumblebee]

This reminds me of the assignment server problems that F@H has. One of my systems is out of work, but 2 others are completely full.

No, no they're not. They are not getting new work either.
As @BUCK NASTY once said, "Good time to clean the systems out."

 

[manofthem]

As @BUCK NASTY once said, "Good time to clean the systems out."

Or in my case, good time to get some unabashed gaming in...and likely strike up a little folding

 

[twilyth]

This is why I keep the max queue of 10 days. But I have noticed that on at least one machine, it goes through several days worth of work in a day or two. But that rig does have connection issues so that might be why. The server doesn't see it for a couple of days and doesn't realize that it's actually been crunching away - which is really sort of dumb since it does end up returning all of the work it was given w/in the time allowed.

 

[twilyth]

It's not looking good gents

And no updates from any of the staff on the WCG site either. I wonder if they got nuked.

 

[manofthem]

It's not looking good gents

And no updates from any of the staff on the WCG site either. I wonder if they got nuked.

Just saw your post over there on the WCG forums. It really would be nice to get some sort of update. I keep checking that thread over there looking for some info but just more of the same so far.

 

[bubbleawsome]

Ah crap. I saw my ppd go to 0, so I readded the project. I have wiped my buffer.
Should have checked the thread

 

[Nordic]

Times like this is why I have a 5 day buffer. My i5's still go through it in 2 days though no matter how much disc space I allocate.

 

[manofthem]

I keep telling myself that my buffer of .2 days just doesn't cut it. My i3 is the only thing with a few more WUs, both i7s are dry.

Anybody else completely dry too?

 

[bubbleawsome]

Yeah.
I blame age of ultron for making all the WCG geeks skip out over the weekend.

 

[[Ion]]

Wonder what's going on--between the no work and the no updates on Free-DC, I wonder if WCG if has suffered some major issues. An announcement would have been nice....

 

[FireFox]

At the end if WCG is not sending works I won't complain because in that case my Machines will have a deserved Brake.

 

[Norton]

WCG is coming back online!

Check your rigs folks, wu's should be flowing again.

I'm on my phone atm, can someone post the links from the WCG site explaining what happened?

 

[twilyth]

http://www.worldcommunitygrid.org/forums/wcg/viewthread_thread,38005_lastpage,yes#lastpost

Here is a digest of Uplinger's posts

1,
Greetings everyone,

I am currently looking into why this has happened. But work is currently being loaded back into the grid as we speak for CEP2, FAHV, MCM1, and UGM. We only load 4 at a time, and OET1 is 5th on the list. It will get work loaded shortly. While the first 4 projects are getting work loaded, members may still get no work available as it is trying to keep up with the demand.

Thanks,
-Uplinger

2.
Yes we should have monitoring for this. We do have monitoring setup for many of the scenarios that happen. In this case the feeder went completely dry, the monitors we have in place test to make sure that the feeder isn't clogged with all resends (an indication that there is a work unit issue or science application issue). So, my main task today will be to update that monitoring to include projects that are completely empty.

Thanks,
-Uplinger

3.
I have just forced the script that creates those reports to run. They should get updated files soon.

Thanks,
-Uplinger

 

[Caring1]

I've got WU's downloading, but very slowly, kb's/sec.
Hopefully they will speed up soon.

 

[manofthem]

I've got WU's downloading, but very slowly, kb's/sec.
Hopefully they will speed up soon.

Yes, very slow downloads right now; I guess they're getting hammered right now. Seems like it's getting ironed out and soon will be flowing nicely again


edit:

Further info from Uplinger

Greetings,

We are extremely sorry about the work unit outage that occurred this weekend. My initial investigation shows that a the scripts that push the workunits into the database was stuck with an illegal lock file. It appears that the lock file was illegal because both servers that attempt to load work happened to create it within the same time. This has been the same mechanism we have used for many years without issue.

I will be adding additional code the the create work scripts to check for an extended period of time and send alerts to our team if the lock file is illegal. I will also see if I can add additional monitoring from an external server to check to make sure there is work available on the servers. At the moment, I need to think what the best possible way of doing this is.

On a personal side, usually I log in and check the grid health every day. Starting thursday of last week, I was spending some time with my twin brother for a long weekend. Since I did not get any alerts due to no checks for the feeder being completely empty as mentioned before, I assumed all was well.

Again, we will be adding some more monitoring so this issue can be caught earlier and work can be consistently flowing. Thank you for your patience with us on this.

Thanks,
-Uplinger

Edit 2: 11am EST and I have some work downloaded to my main rig, finally. Took nearly 2 hrs to get wus dl'd. My 4790 only have 5 wus running, and it's trying to dl more work but still waiting.

 

[Norton]

Thanks for the updates @manofthem and @twilyth !

Looks like we'll get our stats back from FreeDC soon also.... most likely for the midday update. I would expect really strange #'s for a day or two.

 

[Arjai]

OK, got CPUZ for Karen, an E6600 Core 2 Duo 2.4GHz 65W.

Looked up the Gigabyte board and CPU list. Found that the E8600 Core 2 Duo, 3.33GHz 65w, is on the list.

Went to E-Bay, found one for $24. Checked the Bank account, and hit BUY NOW!!

In about 11 days, Karen will be pumping more numbers!! at almost 1 full GHz faster!! All that, for 24 bucks.

Me likey.

 

[peche]

knoxx told me yesterday about this tasks issues that everyone is having, but my 2 i7's still having tasks to do, take a look:
Norad, my first cruncher:

Norad II the new cruncher:

the second one work like a cheetah on cocaine, no stops ever...

 

[thebluebumblebee]

Something's not right. I see WU's pop up in Transfers, they look like they're DL'ing, but nothing's showing up in Tasks. Event log looks like everything's okay. I'm about to run out of work.