Paypal Grinds To A Halt

BillBrasky writes "After a 'Monthly Software Update', it appears that PayPal started having problems. There were reports all weekend of troubles, and as of Monday night here, I can't access it at all (connection time out). One user even reported that his PayPal Debit card was getting refused!" A message on the site now says the site is expected to be back at 8:10 PM PDT, not long from now.

How much negative....

[d3ity]

How much negative feedback will be left on ebay due to user stupidity? The world wonders...

Re:effects on ebay share price tomorrow

[ergo98]

Of course, if the price drops tomorrow, it's probably the best time to buy in, since this is obviously a temporary glitch.

While it may be a temporary glitch, there is no doubt that such an outage has a real financial impact - banking, which PayPal is to a degree, is based upon trust, and PayPal already starts in a precarious situation given that there are no branches to visit and limit options available outside of their website. This is the sort of thing that sends grandmas and grandpas away from that payment method for a while.

Re:effects on ebay share price tomorrow

[ottothecow]

but grandmas and grandpas probobly dont use paypal every day. Even if they check today, they might think they are doing something wrong and call their grandson to walk them through it again. By the time the kids parents make him go help grandpa...paypal will be working fine :)

This is why....

[BobSutan]

"One user even reported that his PayPal Debit card was getting refused!"

This is why real banking institutions have such stringent operational guidelines set down my the federal government in regards to information systems. This should serve as a hearty wake-up call to a great many people that have fallen under the impression that Paypal is a "real" bank, when in fact they are not.

PayPal 101

[xxxJonBoyxxx]

1) As soon as someone pays you, move ALL the money out of your PayPal account.

2) To pay someone else, have payment go directly from your associated checking account. There is no need to carry a positive balance on PayPal.

3) Don't get the goddamn DEBIT card.

BTW, it's 8:37PT and it took almost 3 minutes to sign on. My guess is that PayPal will be hiring soon.

Microsoft's fault

[null etc.]

Isn't it funny how if PayPal was using Windows servers, everyone would be bashing Microsoft right now, blaming it on Windows? (similar to one recently publicized outage)

Hmmm, I wonder what hardware/software they use, and I wonder why no one's bashing it.

Re:This is why....

[Peyna]

great many people that have fallen under the impression that Paypal is a "real" bank

Paypal themselves don't want to be considered a real bank (and the government said they aren't, mostly because they don't physically hold your money), that way they are free from regulation.

Re:That was interesting...

[drinkypoo]

Load testing certainly isn't impossible. Lots of other companies manage to do it.

But paypal isn't a bank, and doesn't guarrantee anything.

Paypal is also doing online banking and handling payments on what seems like an unprecedented scale, very rapidly, and with just about every financial institution in the US. I know banks aren't exactly sitting on their thumbs, especially the larger ones, but they also have the luxury of being able to take a longer view of things than paypal which faces competition capable of putting it out of business should it rest on its laurels too long; such is the nature of the internet.

Maybe someone over there is just incompetent, but I suspect there's real reasons for what happened. I also suspect they'll be looking real hard at how to avoid this happening again. As their relevance increases the public tolerance for this kind of failure will decrease.

I was worried about something like this..

[Another AC]

As soon as I heard ebay had acquired them, I thought "Oh boy, I sure hope ebay doesn't get involved in the technical side of running paypal".

Apparently they have.

Re:Um, yeah...

[chunkwhite86]

Regardless, we accept Paypal payments for our business. Didn't work all weekend...and today I kept getting errors (I think it was 30004) telling me to "retry" or "return to main page". Took a few retries, but I did get stuff done....such as transfering money out for payroll on Friday. Auction and storefront sales were down from lack of a payment service though. Debit card had activity over the weekend, so that worked fine.

You are a moron if you use paypal to run a business including the payroll, of all things. Read the fine print and you'll see that paypal IS NOT A BANK. They don't have to follow bank rules, and your money is NOT FDIC INSURED. Sure, paypal may be fine for processing occational petty payments (Personally, I wouldn't trust them with my bank account or credit card information), but to use it for running a business, including the payroll, should probably be illegal.

On the note of not trusting them. If a new storefront opened in the mall, we'll call them BankPal(tm). They want you to enter your credit card information, bank account numbers, address and phone numbers, and OH BY THE WAY we're not a bank, aren't held to bank standards or laws, can seize your account at any time for no reason, are not fdic insured, blah blah blah, how can anyone with more than two brain cells trust these jokers???

Re:This is why....

[TykeClone]

Then they shouldn't offer the services of a real bank. A paypal debit card, credit card, and money market account sure sound like they are. If they act like a bank, they should be regulated like a bank (are you listening walmart?)

Iiiitttttsssss Baaaaaaaack

[mpechner]

Just completed paying for something I've been trying to pay all day.

Re:not a bank :)

[inflex]

Humor aside, I have to say, banks do crash.

I have a bank run ccard/merchant acceptance interface. About every week I get emails with statements like "Unscheduled down time" or "Delay in payment acceptance" etc. The key difference is that the banks are entirely accountable and the transactions do actually go through ultimately. Furthermore, you have the ability to phone them or visit them personally should you have any issues.

Again, the key here is accountability.

PLD.

Re:It says

[AKAImBatman]

Well, I got in. It looks like PayPal is responding, but REALLY slowly. A couple of points of interest:

1. The front page is LOADED with new graphics and text.

2. There's now a "Powered by Sun" icon on the site.

What I'm wondering is if PayPal didn't overrun their bandwidth with new graphics and pages while simultaneously trying to upgrade the backend systems. The "intermittent problems" might have been caused by such an upgrade. Then Monday hits and they haven't completed upgrading/stress testing the system. What happens? Nose dive!

If my theory is correct, we can expect the performance to slowly improve as PayPal gets more systems online. Also keep an eye on that front page to see if any sudden changes show up.

Re:hmm...

[Anonymous Coward]

This is PayPal. A claim like that will just make them suspect you of fraud, and they'll confiscate whatever is in your account.

Re:PayPal 101

[xxxJonBoyxxx]

Charging to a checking account is just downright unethical.

Hmmm...ever try online bill pay?

We did get the money back - 3 weeks later.

So what's the problem? If no one helped...how did the money get returned?

I agree anyone who ties their PayPal account to their main checking account is a moron. However, these days free checking accounts are available at almost any bank. Set up a dedicated checking account for PayPal at a local branch of your favorite bank and you're in business. Think of it as your own financial firewall.

Rollback plan?

[bigtangringo]

That's why people make rollback plans. If I had the budget of paypay I'd have my new sun servers ready, points my VIPs to the new sun servers and if everything goes to hell, go back to the old VIPs and back to the drawing board.

Psh.. amateurs :)

TILT

[Doc Ruby]

Yet another reason this unregulated global banking monopoly has to get some competition, immediately. The ecommerce world cannot afford to rely on this single point of failure, either accidentally or at their discretion.

Re:That was interesting...

[HotshotXV]

You think that Paypal is doing business on an unprecedented scale? Sure, maybe in the U.S., banks don't handle as much of a load, seeing as there are a million different branches... but look at credit card companie, and how there is never any downtime in their systems. You think that VISA's billing department, or the credit verification department, or many others don't do the same amount of business as PayPal? (if not more) Most of the comments are right - you don't try to implement a new interface or a new system without extensive load testing... If you can't do extensive load testing, you have to either phase things in VERY slowly, or scrap the update until the testing can be done.

Rollback? Fallback plan?

[Anonymous Coward]

I'm surprised nobody has mentioned it. But I recall, most large operations, having an "undo" plan of action, so if the upgrade fails, or has problems, there's a way to migrate back down to the software you were running before.

It would make more sense to rollback, regain reliable service, and re-test the failed upgrade on their internal test network.

Re:PayPal can't be down!

[HotshotXV]

Is it not a little frightening that the Slashdot users, the so-called "tech elite", can still come close to falling for these scams? What hope does the home user who isn't even adept enough to use more than two fingers to type have? Maybe that's the business I should get in...

Re:It says

[xxxJonBoyxxx]

I'm wondering is if PayPal didn't overrun their bandwidth with new graphics and pages...

I would hope we couldn't blame serving up static content as a cause; there are so many cache servers built for this task in 2004 that one would hope actual web servers never actually see these requests.

If my theory is correct, we can expect the performance to slowly improve as PayPal gets more systems online

Nah...if it's bandwidth-related, performance will take another hit as more systems come online and try to share a saturated pipe.

However, if the problems WERE bandwidth related, it should be easy for eBay to diagnose; just take a meter reading of traffic before the crappy performance and another after. Likewise, it should be easy for eBay to diagnose server performance problems - just look for the spike.

I think what is troubling to every PayPal user is not that PayPal/eBay is having problems, but that they appear to have no idea what is broken and no idea how to fix it.

Re:It says

[MyThoughts]

Nah...if it's bandwidth-related, performance will take another hit as more systems come online and try to share a saturated pipe.

Do you actually think PayPal runs their servers over one pipe? They probably have servers across the globe serving up the site. Adding more systems online (across the globe) would improve performance. But, one would hope, they know enough about what they are doing to not let a bandwidth miscalculation put them offline.

Re:How about these escrow services? They're still

[HotshotXV]

Of course, these services are all inferior to PayPal because of the exorbant fees that each charges. Useful in a pinch, but if Paypal disappeared, I would wager that most merchants would choose to set up a merchant account directly with CC companies, and avoid the extra cost. The convenience just isn't worth it.

How will you know the difference?

[Anonymous Coward]

How much negative feedback will be left on ebay due to user stupidity? The world wonders...

And this will be different how?

Re:advantages of Paypal not being a bank...

[dr_d_19]

There are a lot of people, for a variety of reasons, who have been screwed out of getting a bank account from a regular bank. Because of negative reporting on chex systems, they can't get a regular checking account at all.

Yeah, they can be as happy as they want, but I for one am VERY happy that my bank checks up on other customers, and there are cases when it's obvious that PayPal should have done the same.

Re:PayPal and eBay constantly push the envelope

[coene]

The "user systems" and associated features of these bank sites have nowheres near the relational properties of something like PayPal - it's not even close. They can segregate data based on customer sets, where you know where users are coming from, and you can tie them to their own set of data - data that can be archived and summarized into non-realtime aggregates.

PayPal has the task of linking transactions between users, and allowing users to create modifications or additions to them in real time.

These are not the same, by any stretch. It's the final bit of difference that makes it so much more complex.

Upgrades are always a nightmare - some thoughts

[MarkH]

1) You should always have a detailed rollback plan. This should include what you monitor during the change, what triggers the rollback and who is responsible for making that decision

2) You should have a rollback plan for any change to a live site which deals with transactions no matter how 'small' the site is

The problem these days is the complex interaction between different systems and user traffic means there is no guarantee of a flawless rollout even with a full staging system to test on. I have seen failures occuring due to a unique combination of user requests creating a thread contention which results in an instability causing failure up to 1/2 an hour later!

There is a gap for a system which can capture the entire network traffic being applied to a system and replay it against a staging environment to catch just this sort of problem.

The other problem is a 'minor' problem during rollout such as intermittant failure of some parts. This is let go for a while and people start throwing hacks at the problem. Before you know it enought new transactions have been applied to the system to make rollback unviable.

In my opinion with complex web based systems testing and rollout is becoming one of the most interesting areas in IT at the moment

Re:That was interesting...

[jrockway]

That's why I always pay PayPal with my credit card. Something goes wrong? *chargeback* Done. It's against their ToS, but they can lick my balls. It's my money and I'm not going to cry if they're not allowed to steal it.

They can sue me for all $200 I possess. :-)

Re:hmm...

[tempmpi]

Shutdown maybe but I don't believe they kept the money.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Why? real banks have outages too.

[SuperKendall]

I've had a number of online banks in the past, there is not one that has not had an outage at some point or another - even regular Sunday night outages in some cases.

PayPal in my mind has a better track record than most real banks I've used! As you say, they have to, but they also will not be penalized much for a small glitch like this.

They also have the advantage that there is NOTHING else like PayPal. I've looked at other options and there are none. So where are you gonna go? I've tried two auctions in the past that tried to use non-PayPay payments, and all I can say is - never again.